[Paper Review] Semi-supervised Sequence Learning

Andrew M. Dai, Quoc V. Le|arXiv (Cornell University)|Nov 4, 2015

Topic Modeling38 references490 citations

TL;DR

This paper proposes semi-supervised pretraining for recurrent neural networks using language modeling and sequence autoencoders to improve training stability and generalization in sequence classification tasks. By initializing LSTMs with unsupervised objectives on large unlabeled data, the method achieves state-of-the-art or competitive results on IMDB, DBpedia, 20 Newsgroups, and CIFAR-10 without additional labeled data.

ABSTRACT

We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.

Motivation & Objective

• To address the instability and poor generalization of training deep LSTM networks for sequence classification tasks.
• To explore unsupervised pretraining techniques that leverage large amounts of unlabeled data to improve supervised sequence learning.
• To evaluate whether pretraining with autoencoders or language models leads to better performance and faster convergence than random initialization.
• To investigate the transferability of pretraining across related domains, such as using Amazon reviews to improve Rotten Tomatoes sentiment classification.
• To extend the applicability of semi-supervised sequence learning beyond text to non-sequential data, such as images processed row-by-row.

Proposed method

• Train a recurrent language model (next-token prediction) on unlabeled sequences to pretrain LSTM weights.
• Implement a sequence autoencoder that encodes an input sequence into a fixed-length vector and reconstructs the original sequence using shared encoder-decoder weights.
• Use the learned encoder weights from the autoencoder or language model as initialization for a downstream supervised LSTM classifier.
• Apply linear label gain by gradually increasing the loss weight on the label prediction from 0 to 1 across timesteps to improve gradient flow to earlier layers.
• Fine-tune the preinitialized LSTM on labeled data using standard backpropagation through time.
• Extend the method to non-text data by processing images row-by-row as sequences, using L2 loss for reconstruction in the autoencoder.

Experimental results

Research questions

• RQ1 Can unsupervised pretraining with a language model or sequence autoencoder improve the training stability and generalization of LSTMs on text classification tasks?
• RQ2 Does pretraining on large-scale unlabeled data from related domains (e.g., Amazon reviews) improve performance on a downstream classification task (e.g., Rotten Tomatoes) without additional labeled data?
• RQ3 How does the combination of pretraining and linear label gain affect performance on long sequences and deep LSTMs?
• RQ4 Can the proposed semi-supervised approach generalize to non-textual sequential data, such as images processed as pixel-row sequences?
• RQ5 Is the performance of pre-trained LSTMs competitive with or superior to state-of-the-art models on standard benchmarks like IMDB, DBpedia, and CIFAR-10?

Key findings

• SA-LSTM and LM-LSTM models outperform randomly initialized LSTMs across all benchmarks, with SA-LSTM achieving 15.6% test error on 20 Newsgroups, a significant improvement over previous methods.
• Using unlabeled Amazon review data to pretrain the sequence autoencoder improved Rotten Tomatoes sentiment classification accuracy from 79.7% to 83.3%, equivalent to adding substantial labeled data.
• On the DBpedia character-level classification task, the SA-LSTM with linear gain achieved 1.19% test error, surpassing state-of-the-art convolutional networks (e.g., 1.73% for Large ConvNet).
• The linear label gain mechanism reduced test error from 1.32% (LSTM with linear gain) to 1.19% when combined with SA-LSTM on DBpedia, demonstrating improved gradient flow.
• On CIFAR-10, the 2-layer LM-LSTM achieved 18.0% test error, outperforming the non-pretrained LSTM (26.0%) and the baseline convolutional DBN (21.1%).
• The sequence autoencoder method was more stable during training than standard LSTMs, and the pretraining step enabled training of LSTMs on sequences up to several hundred timesteps.