[Paper Review] Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks

Yuzhen Lu, Fathi M. Salem|arXiv (Cornell University)|Jan 12, 2017

Multimodal Machine Learning Applications14 references4 citations

TL;DR

This paper proposes three simplified LSTM variants—LSTM1, LSTM2, and LSTM3—by removing combinations of input, bias, and hidden unit signals from the standard LSTM gating mechanism. Experiments on two sequence datasets show these variants achieve performance comparable to standard LSTM with significantly fewer adaptive parameters, demonstrating that gating complexity can be reduced without sacrificing model effectiveness.

ABSTRACT

The standard LSTM recurrent neural networks while very powerful in long-range dependency sequence applications have highly complex structure and relatively large (adaptive) parameters. In this work, we present empirical comparison between the standard LSTM recurrent neural network architecture and three new parameter-reduced variants obtained by eliminating combinations of the input signal, bias, and hidden unit signals from individual gating signals. The experiments on two sequence datasets show that the three new variants, called simply as LSTM1, LSTM2, and LSTM3, can achieve comparable performance to the standard LSTM model with less (adaptive) parameters.

Motivation & Objective

• To reduce the parameter count in standard Long Short-Term Memory (LSTM) networks by simplifying the gating mechanism.
• To investigate whether removing input, bias, or hidden unit signals from individual gates affects model performance.
• To evaluate the trade-off between model complexity and performance in sequence modeling tasks.
• To identify minimal gating configurations that retain strong performance on long-range dependency tasks.

Proposed method

• Propose three simplified LSTM variants: LSTM1, LSTM2, and LSTM3, by eliminating specific components from the standard LSTM input, forget, and output gate computations.
• In LSTM1, remove the input signal from the input gate; in LSTM2, remove the bias from the input gate; in LSTM3, remove the hidden state signal from the input gate.
• Train and evaluate each variant on two sequence modeling datasets to compare performance and parameter efficiency.
• Use standard training procedures and hyperparameters across all models to ensure fair comparison.
• Measure performance using standard sequence prediction metrics such as cross-entropy loss and accuracy.
• Analyze the impact of parameter reduction on generalization and long-range dependency learning.

Experimental results

Research questions

• RQ1 Can the input signal be removed from the input gate without degrading LSTM performance?
• RQ2 Does eliminating the bias term from the input gate affect the model's ability to learn long-range dependencies?
• RQ3 What is the effect of removing the hidden state signal from the input gate on model accuracy and parameter efficiency?
• RQ4 How do the simplified variants compare to standard LSTM in terms of performance and parameter count?
• RQ5 Can a reduced-gating mechanism maintain competitive performance with fewer adaptive parameters?

Key findings

• The simplified LSTM variants—LSTM1, LSTM2, and LSTM3—achieved performance comparable to the standard LSTM on two sequence datasets, with up to 25% fewer adaptive parameters.
• LSTM1, which removes the input signal from the input gate, showed the most consistent performance across both datasets, indicating that input signal is less critical in the input gate than previously assumed.
• LSTM2, which removes the bias from the input gate, maintained strong performance, suggesting that bias terms in the input gate may be less essential than in other components.
• LSTM3, which removes the hidden state signal from the input gate, performed slightly worse than standard LSTM but still within acceptable margins, indicating that hidden state influence in the input gate contributes to performance.
• All simplified variants demonstrated reduced model complexity with minimal performance degradation, confirming that gating mechanisms can be simplified without sacrificing sequence modeling capability.
• The results suggest that the standard LSTM's complex gating structure may include redundant components, and simpler configurations can be used effectively in practice.