[Paper Review] Simplified Long Short-term Memory Recurrent Neural Networks: part II

Atra Akandeh, Fathi M. Salem|arXiv (Cornell University)|Jul 14, 2017

Neural Networks and Applications4 citations

TL;DR

This paper proposes six simplified Long Short-Term Memory (LSTM) variants—LSTM4, LSTM5, LSTM4a, LSTM5a, and LSTM6—by drastically reducing adaptive parameters through fixed or vectorized gating mechanisms. The models achieve comparable MNIST test accuracy to the standard LSTM while reducing parameters by up to 74% and training time per epoch by 60%, with sustained performance under high learning rates when using ReLU activation, unlike standard LSTM which degrades.

ABSTRACT

This is part II of three-part work. Here, we present a second set of inter-related five variants of simplified Long Short-term Memory (LSTM) recurrent neural networks by further reducing adaptive parameters. Two of these models have been introduced in part I of this work. We evaluate and verify our model variants on the benchmark MNIST dataset and assert that these models are comparable to the base LSTM model while use progressively less number of parameters. Moreover, we observe that in case of using the ReLU activation, the test accuracy performance of the standard LSTM will drop after a number of epochs when learning parameter become larger. However all of the new model variants sustain their performance.

Motivation & Objective

• To reduce the computational cost and training time of standard LSTM models by minimizing adaptive parameters.
• To evaluate the performance of parameter-reduced LSTM variants on the MNIST benchmark dataset.
• To investigate the stability and accuracy of simplified LSTMs under varying learning rates and activation functions.
• To determine whether simplified LSTMs can match standard LSTM performance with significantly reduced complexity.
• To explore the impact of fixed gate values on model stability and generalization.

Proposed method

• Proposes LSTM4 and LSTM5 by replacing full recurrent weight matrices in the input, forget, and output gates with learnable vectors, enabling pointwise multiplication.
• Introduces LSTM4a and LSTM5a by fixing the forget gate to 0.96 and output gate to 1.0, eliminating adaptive parameters in these gates.
• Introduces LSTM6 by setting all gates to constant values (forget=0.59, input=1.0, output=1.0), reducing the model to a basic RNN with minimal parameters.
• Uses the Keras library to train and evaluate all variants on the MNIST dataset using sequence-based row-wise input (28x28 images as 28-step sequences).
• Employs three activation functions—tanh, sigmoid, and ReLU—and tunes the learning rate (η) across three values to assess robustness.
• Compares model performance via training and test accuracy, parameter count, and time per epoch across all variants.

Experimental results

Research questions

• RQ1 Can simplified LSTM variants with fewer adaptive parameters achieve comparable test accuracy to the standard LSTM on the MNIST dataset?
• RQ2 How does fixing gate values (e.g., forget gate to 0.96 or 0.59) affect model stability and performance?
• RQ3 Does the choice of activation function (tanh, sigmoid, ReLU) influence the performance and convergence of simplified LSTMs?
• RQ4 How do different learning rates impact the training dynamics of simplified LSTMs versus the standard LSTM?
• RQ5 Can simplified LSTMs maintain performance under high learning rates where standard LSTM degrades?

Key findings

• The standard LSTM model's test accuracy drops significantly under high learning rates (η=2e-3) when using ReLU activation, while all simplified variants maintain stable performance.
• LSTM5 achieves the highest test accuracy among simplified variants (0.9892) with ReLU activation and η=1e-4, closely approaching the standard LSTM’s 0.9853.
• LSTM6, with all gates fixed to constants, achieves 96.56% test accuracy with ReLU and η=2e-3, demonstrating that a basic RNN can perform competitively when properly tuned.
• LSTM4a and LSTM5a show the most consistent performance across different activation functions and learning rates, with test accuracies of 97.92% and 98.21% respectively under ReLU and η=2e-3.
• The number of parameters is reduced from 52,610 in the standard LSTM to 13,910 in LSTM6, a 74% reduction, while training time per epoch is reduced by 60%.
• All simplified variants maintain high performance under high learning rates when using ReLU, unlike the standard LSTM, which suffers performance degradation after ~50 epochs.