[Paper Review] Unsupervised Learning of Video Representations using LSTMs

Nitish Srivastava, Elman Mansimov+1 more|arXiv (Cornell University)|Feb 16, 2015

Human Pose and Action Recognition30 references1,663 citations

TL;DR

This paper proposes an unsupervised LSTM-based autoencoder and future prediction model to learn video representations from untrimmed video sequences. By training on YouTube videos without labels, the model learns disentangled, generalizable features that improve action recognition accuracy—especially with few labeled examples—demonstrating strong transfer learning performance on UCF-101 and HMDB-51.

ABSTRACT

We use multilayer Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We try to visualize and interpret the learned features. We stress test the model by running it on longer time scales and on out-of-domain data. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only a few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

Motivation & Objective

• To learn meaningful, disentangled video representations in an unsupervised manner using temporal sequence modeling.
• To evaluate whether unsupervised pretraining with LSTMs improves performance on downstream supervised action recognition tasks.
• To investigate the impact of different training objectives—reconstruction vs. future prediction—on representation quality.
• To analyze the generalization and extrapolation capabilities of learned representations beyond training time scales.
• To assess transferability of representations learned on unrelated video data (e.g., 300 hours of YouTube) to action recognition benchmarks.

Proposed method

• Uses a multilayer LSTM encoder to compress a sequence of video frames into a fixed-length latent representation.
• Employs one or more decoder LSTMs to reconstruct the input sequence or predict future frames from the encoded representation.
• Trains the model using two primary objectives: autoencoding (reconstruction) and future prediction, with a composite model combining both.
• Uses two input types: raw image patches (e.g., MNIST digits) and high-level percepts from a pretrained ImageNet convolutional net.
• Applies conditional decoding by feeding generated outputs back into the decoder, comparing performance with and without conditioning.
• Evaluates representations by fine-tuning the encoder on supervised action recognition tasks using UCF-101 and HMDB-51 datasets.

Experimental results

Research questions

• RQ1 Can an unsupervised LSTM-based model learn generalizable video representations that capture motion and appearance structure without labels?
• RQ2 How does combining reconstruction and future prediction objectives affect the quality of learned representations compared to using either alone?
• RQ3 To what extent can representations pretrained on unrelated video data (e.g., YouTube) improve performance on action recognition with limited labeled examples?
• RQ4 How well do the models extrapolate motion and appearance beyond the training sequence length?
• RQ5 Does conditioning the decoder on its own generated outputs improve the quality of future predictions or representation learning?

Key findings

• The composite model, combining both autoencoding and future prediction objectives, achieved the best performance on action recognition, reaching 75.8% accuracy on UCF-101 and 44.0% on HMDB-51.
• Pretraining on 300 hours of YouTube videos significantly improved action recognition accuracy, especially when only a few labeled examples were available.
• The model demonstrated persistent generation of plausible motion sequences beyond the training time scale, though object details degraded over long horizons.
• Conditioning the decoder on its own output did not significantly improve supervised performance but led to slightly better qualitative future predictions.
• The model outperformed a standard LSTM baseline and matched or exceeded state-of-the-art models using only RGB data, including LRCN and C3D.
• Combining RGB and flow-based predictions achieved 84.3% accuracy on UCF-101, showing strong potential for fusion with other modalities.