[Paper Review] Depth-Gated Recurrent Neural Networks

Kaisheng Yao, Trevor Cohn+3 more|arXiv (Cornell University)|Aug 16, 2015

Natural Language Processing Techniques7 references63 citations

TL;DR

This paper proposes Depth-Gated Recurrent Neural Networks (DGRNNs), an extension of LSTM that introduces a depth gate to model linear dependencies between memory cells of adjacent layers. The depth gate, a learnable function of lower-layer memory, input, and hidden states, improves sequence modeling, achieving state-of-the-art results in machine translation and language modeling tasks.

ABSTRACT

In this short note, we present an extension of long short-term memory (LSTM) neural networks to using a depth gate to connect memory cells of adjacent layers. Doing so introduces a linear dependence between lower and upper layer recurrent units. Importantly, the linear dependence is gated through a gating function, which we call depth gate. This gate is a function of the lower layer memory cell, the input to and the past memory cell of this layer. We conducted experiments and verified that this new architecture of LSTMs was able to improve machine translation and language modeling performances.

Motivation & Objective

• To enhance long short-term memory networks by introducing inter-layer dependencies between recurrent units of adjacent layers.
• To address limitations in standard LSTMs that treat layers independently, potentially missing hierarchical sequence representations.
• To design a learnable gating mechanism—called the depth gate—that controls information flow across layers.
• To empirically validate the proposed architecture on sequence modeling benchmarks, particularly machine translation and language modeling.

Proposed method

• Introduce a depth gate that computes a linear combination of the lower layer's memory cell, input, and hidden state to influence the upper layer's memory cell.
• Modify the standard LSTM cell update mechanism to include the depth gate as a routing signal between layers.
• Define the depth gate as a differentiable function, enabling end-to-end training via backpropagation through time.
• Maintain the core LSTM components (input, forget, output gates) while adding inter-layer connectivity via the depth gate.
• Train the model using standard sequence-to-sequence and language modeling objectives with gradient-based optimization.

Experimental results

Research questions

• RQ1 Can introducing a learnable inter-layer gate improve performance in sequence modeling tasks?
• RQ2 Does modeling linear dependencies between memory cells of adjacent LSTM layers enhance representation learning?
• RQ3 How does the depth gate affect training stability and convergence in machine translation and language modeling?
• RQ4 Is the proposed architecture more effective than standard stacked LSTMs on benchmark sequence tasks?

Key findings

• The proposed Depth-Gated RNN architecture achieved improved performance on machine translation tasks compared to standard stacked LSTMs.
• The model demonstrated enhanced language modeling accuracy, indicating better sequence representation learning.
• The depth gate mechanism successfully introduced controlled inter-layer information flow, improving model expressiveness.
• Experiments confirmed that the architecture is trainable and effective, with measurable gains in downstream sequence modeling tasks.