[Paper Review] The unreasonable effectiveness of the forget gate

Jos van der Westhuizen, Joan Lasenby|arXiv (Cornell University)|Apr 13, 2018

Low-power high-performance VLSI design39 references71 citations

TL;DR

JANET, a forget-gate-only LSTM variant with chrono initialization, matches or surpasses standard LSTM performance on MNIST, permuted MNIST, and MIT-BIH datasets while offering computational savings.

ABSTRACT

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.

Motivation & Objective

• Investigate whether all LSTM gates are necessary by evaluating a forget-gate-only architecture.
• Assess the performance of the JANET against standard LSTM and other RNN variants on benchmark datasets.
• Explain the role of chrono initialization in training stability and memory retention.
• Quantify theoretical computational and memory savings of JANET over LSTM.

Proposed method

• Derive JANET by removing input and output gates from LSTM and coupling input/forget modulation (f_t and c_t updates).
• Remove tanh on h_t to avoid unnecessary gradient attenuation and apply optional beta-based offset to accentuate information accumulation.
• Apply chrono initialization to forget and input gates to control forgetting time constants based on T_max.
• Provide a theoretical gradient analysis comparing JANET and LSTM to explain training ease and gradient flow. 0
• Compute approximate hardware savings: parameter counts, memory footprint, and forward-pass compute, assuming comparable accuracy to LSTM.

Experimental results

Research questions

• RQ1 Is a forget-gate-only architecture (JANET) capable of matching or surpassing standard LSTM performance across diverse tasks?
• RQ2 Do chrono initialization schemes improve training stability and memory retention for JANET/LSTM variants?
• RQ3 What are the practical computational and memory savings when replacing LSTM with JANET in forward passes?
• RQ4 How do JANET and LSTM compare in gradient propagation and optimization difficulty on standard benchmarks?

Key findings

• JANET achieves 99.0% on MNIST, 92.5% on pMNIST, and 89.4% on MIT-BIH, outperforming the standard LSTM which achieves 98.5%, 91.0%, and 87.4%, respectively.
• Reducing gates to a forget gate with coupling and removing the h_t nonlinearity yields competitive or superior accuracy across datasets.
• The JANET architecture enables skip-like connections over time, contributing to easier and faster training compared to the LSTM.
• The JANET has roughly half as many parameters as the LSTM, and forward-pass computation is estimated at about 5/6 of the LSTM’s, implying hardware efficiency gains.
• Chrono initialization of forget (and opposite) gate biases mitigates memory retention problems and aids training on longer sequences (e.g., MNIST subsequences).
• On larger layer sizes and under chrono initialization, JANET narrows or matches the performance gap to top models like WaveNet on pMNIST.