[Paper Review] Densely Connected Bidirectional LSTM with Applications to Sentence Classification

Zixiang Ding, Rui Xia+3 more|arXiv (Cornell University)|Feb 3, 2018

Topic Modeling19 references6 citations

TL;DR

This paper proposes Densely Connected Bidirectional Long Short-Term Memory (DC-Bi-LSTM), a deep RNN architecture that enhances feature reuse and information flow by concatenating hidden states from all previous layers. The model achieves significant accuracy gains over standard Bi-LSTM and state-of-the-art methods on five sentence classification benchmarks, even with fewer parameters, and trains successfully up to 20 layers despite vanishing gradient issues.

ABSTRACT

Deep neural networks have recently been shown to achieve highly competitive performance in many computer vision tasks due to their abilities of exploring in a much larger hypothesis space. However, since most deep architectures like stacked RNNs tend to suffer from the vanishing-gradient and overfitting problems, their effects are still understudied in many NLP tasks. Inspired by this, we propose a novel multi-layer RNN model called densely connected bidirectional long short-term memory (DC-Bi-LSTM) in this paper, which essentially represents each layer by the concatenation of its hidden state and all preceding layers' hidden states, followed by recursively passing each layer's representation to all subsequent layers. We evaluate our proposed model on five benchmark datasets of sentence classification. DC-Bi-LSTM with depth up to 20 can be successfully trained and obtain significant improvements over the traditional Bi-LSTM with the same or even less parameters. Moreover, our model has promising performance compared with the state-of-the-art approaches.

Motivation & Objective

• To address vanishing gradients and overfitting in deep stacked RNNs for NLP tasks.
• To improve feature reuse and information flow in deep bidirectional LSTM architectures.
• To enable successful training of very deep RNNs (up to 20 layers) for sentence classification.
• To achieve better performance and parameter efficiency than standard Bi-LSTM and state-of-the-art models.

Proposed method

• The model uses a densely connected architecture where each layer's input is the concatenation of the original input sequence and all preceding layer hidden states.
• Each layer processes the input via a bidirectional LSTM, producing a new hidden state representation.
• Hidden states from all previous layers are concatenated and passed forward to subsequent layers, enabling deep feature reuse.
• The final layer's hidden state is used as the sentence-level representation for classification.
• The architecture is trained end-to-end with standard backpropagation and gradient descent.
• Hyperparameters such as depth (dl), number of hidden units (dh), and sequence length (th) are tuned for optimal performance.

Experimental results

Research questions

• RQ1 Can a densely connected architecture alleviate vanishing gradient and overfitting issues in deep bidirectional LSTMs?
• RQ2 Does concatenating all previous layer hidden states improve feature representation and classification accuracy?
• RQ3 Can DC-Bi-LSTM be successfully trained at depths of 20 or more layers, unlike standard stacked Bi-LSTMs?
• RQ4 How does DC-Bi-LSTM compare in performance and parameter efficiency to standard Bi-LSTM and state-of-the-art models?

Key findings

• DC-Bi-LSTM with 20 layers achieves 50.2% accuracy on SST-1 and 88.8% on SST-2, outperforming the standard Bi-LSTM baseline (49.2% and 87.2%) with the same or fewer parameters.
• The model with 15 layers (dl=15, dh=13) achieved the best performance on SST-1 (51.9%) and SST-2 (89.7%), indicating an optimal depth for this setup.
• When parameter count was fixed at 1.44M, DC-Bi-LSTM with 10 layers (dl=10, dh=20) achieved 51.0% on SST-1 and 88.5% on SST-2, showing better parameter efficiency than Bi-LSTM.
• Increasing depth (dl) from 0 to 20 consistently improved performance on both SST-1 and SST-2, with gains of 1.7% and 1.3% respectively, demonstrating the benefit of deeper architectures.
• Increasing the number of hidden units (dh) from 0 to 20 led to a steady improvement in accuracy, with 51.0% on SST-1 and 88.5% on SST-2 at dh=20, confirming the importance of representational capacity per layer.
• The model remained stable and trainable at depths up to 20, whereas standard stacked LSTM performance dropped sharply beyond 5 layers (e.g., 30% accuracy at depth 8), confirming its training advantages.