[Paper Review] Recurrent Highway Networks

Julian Zilly, Rupesh K. Srivastava+2 more|arXiv (Cornell University)|Jul 12, 2016

Natural Language Processing Techniques32 references86 citations

TL;DR

This paper introduces Recurrent Highway Networks (RHNs), a novel RNN architecture that increases recurrence depth beyond one by stacking Highway layers within the recurrent transition, enabling deeper, more powerful sequential modeling. Using a theoretical analysis based on Gerçgorin's circle theorem, the authors show RHNs mitigate vanishing/exploding gradients and achieve state-of-the-art results: on Penn Treebank, perplexity drops from 90.6 to 65.4 by increasing recurrence depth from 1 to 10 with the same parameters, and on enwik8, RHNs achieve 1.27 bits per character, outperforming prior methods.

ABSTRACT

Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.

Motivation & Objective

• To address the challenge of training deep recurrent networks with long credit assignment paths in time and space.
• To improve upon existing RNN architectures like LSTMs and stacked RNNs, which struggle with vanishing/exploding gradients when increasing depth.
• To develop a theoretically grounded method for increasing recurrence depth that enhances modeling capacity without sacrificing training stability.
• To demonstrate that deeper recurrent transitions via Highway layers lead to significantly better performance on standard language modeling benchmarks.

Proposed method

• Proposes Recurrent Highway Networks (RHNs), which extend the LSTM architecture by replacing the single recurrent transition with a stack of Highway layers, enabling recurrence depth greater than one.
• Applies Gerçgorin's circle theorem to analyze gradient flow in RNNs, revealing insights into how gates control error backpropagation and why Highway connections stabilize training.
• Uses Highway layers with learnable transform and carry gates to enable adaptive, dynamic computation across multiple nonlinear transformations per time step.
• Employs layer normalization (LN) to stabilize training and improve generalization, particularly in deep recurrent transitions.
• Designs the RHN architecture to allow each time step to perform multiple nonlinear transformations, effectively increasing the depth of the recurrence path.
• Optimizes the network using standard backpropagation through time, with weight tying and dropout for regularization.

Experimental results

Research questions

• RQ1 Can increasing the recurrence depth in RNNs beyond one layer significantly improve modeling capacity and performance on sequential tasks?
• RQ2 How do Highway layers in the recurrent transition affect gradient flow and training stability compared to standard RNNs or stacked RNNs?
• RQ3 What is the theoretical basis for the improved gradient dynamics in RHNs, and how does it relate to the behavior of LSTM cells?
• RQ4 To what extent does increasing recurrence depth improve performance on language modeling benchmarks like Penn Treebank and enwik8?
• RQ5 How do the individual Highway layers in the recurrent transition contribute to overall network performance?

Key findings

• On the Penn Treebank dataset, increasing recurrence depth from 1 to 10 in RHNs reduced word-level perplexity from 90.6 to 65.4 using the same number of parameters.
• On the enwik8 dataset, RHNs achieved a test set entropy of 1.27 bits per character, outperforming all previous methods.
• On the text8 dataset, RHNs achieved a test set entropy of 1.27 bits per character with a model size of 45M parameters, setting a new state-of-the-art.
• The lesioning experiment showed that the first layer in the recurrent transition contributes most to performance, but removing any layer significantly degrades results due to the recurrent structure.
• RHNs with recurrence depth 10 achieved 68.5 BPC on enwik8 and 65.4 perplexity on Penn Treebank, outperforming models like Variational LSTM and Neural Architecture Search-based models.
• Theoretical analysis using Gerçgorin's circle theorem revealed that Highway layers stabilize gradient flow by controlling the spectral radius of the temporal Jacobian matrix, explaining their success in deep RNNs.