[Paper Review] Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Trieu H. Trinh, Andrew M. Dai+2 more|arXiv (Cornell University)|Mar 1, 2018

Topic Modeling33 references87 citations

TL;DR

The paper introduces unsupervised auxiliary losses added to RNNs (r-LSTM and p-LSTM) to reconstruct or predict subsequences, enabling longer-term dependency learning with truncated BPTT and improved efficiency, outperforming several baselines on long sequences.

ABSTRACT

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective. This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel image classification with sequence lengths up to 16\,000, and a real document classification benchmark. Our results highlight good performance and resource efficiency of this approach over competitive baselines, including other recurrent models and a comparable sized Transformer. Further analyses reveal beneficial effects of the auxiliary loss on optimization and regularization, as well as extreme cases where there is little to no backpropagation.

Motivation & Objective

• Address the challenge of learning long-term dependencies in sequences with backpropagation through time (BPTT) and high memory costs.
• Propose auxiliary unsupervised losses that reconstruct past subsequences or predict future subsequences at random anchor points.
• Show that these auxiliary losses enable effective training with truncated BPTT and improve optimization and generalization.
• Evaluate on very long sequence tasks (up to 16,000 steps) and compare with recurrent baselines and Transformer variants.

Proposed method

• Introduce random anchor points in the input sequence.
• Add a reconstruction auxiliary loss that reconstructs a past subsequence from the anchor point.
• Add a prediction auxiliary loss that predicts a future subsequence from the anchor point.
• Train with two phases: unsupervised pretraining of the auxiliary loss, then semi-supervised training combining the main supervised loss with the auxiliary loss.
• Use truncated backpropagation to keep BPTT cost constant across sequence lengths.
• Evaluate with r-LSTM (reconstruction) and p-LSTM (prediction) on long sequences and compare with LSTM baselines and Transformer.

Experimental results

Research questions

• RQ1 Can unsupervised auxiliary losses improve learning of long-range dependencies in RNNs when using truncated BPTT?
• RQ2 Do reconstruction and prediction auxiliary losses provide optimization or regularization benefits for long sequence modeling?
• RQ3 How do r-LSTM and p-LSTM compare to LSTMs and Transformers on long sequence benchmarks in terms of accuracy and efficiency?
• RQ4 What are the effects of sampling frequency and subsequence length on auxiliary loss effectiveness?
• RQ5 How scalable are these methods to very long sequences (up to 16,000 steps) and different data domains (images, text)?

Key findings

• Auxiliary losses enable strong performance with truncated BPTT, matching or approaching fully backpropagated RNNs on several tasks.
• On MNIST, pMNIST, CIFAR10, and StanfordDogs with long sequences, r-LSTM and p-LSTM outperform fully trained LSTMs under truncation and show better scalability as sequence length grows.
• For long sequences, r-LSTM and p-LSTM demonstrate substantial computational efficiency, with training times remaining manageable where full BPTT becomes infeasible.
• In DBpedia character-level classification, auxiliary losses with truncated BPTT outperform LSTMs and other baselines, including SA-LSTM and LM-LSTM, by notable margins.
• Compared to Transformer baselines, Transformers may achieve higher accuracy on shorter sequences, but performance degrades with longer sequences, while r-LSTM/p-LSTM remain robust and memory-efficient.