[Paper Review] Unsupervised Learning of Visual Structure using Predictive Generative Networks

William Lotter, Gabriel Kreiman+1 more|arXiv (Cornell University)|Nov 19, 2015

Advanced Vision and ImagingComputer Science34 references82 citations

TL;DR

This paper proposes a CNN-LSTM-deCNN architecture trained to predict future video frames using a predictive loss, demonstrating that such unsupervised training leads to rich, disentangled representations of underlying 3D object structure. Despite being trained only on pixel-level prediction, the model learns transformation-tolerant features that generalize well to downstream tasks like static image classification, outperforming models trained with reconstruction losses.

ABSTRACT

The ability to predict future states of the environment is a central pillar of intelligence. At its core, effective prediction requires an internal model of the world and an understanding of the rules by which the world changes. Here, we explore the internal models developed by deep neural networks trained using a loss based on predicting future frames in synthetic video sequences, using a CNN-LSTM-deCNN framework. We first show that this architecture can achieve excellent performance in visual sequence prediction tasks, including state-of-the-art performance in a standard 'bouncing balls' dataset (Sutskever et al., 2009). Using a weighted mean-squared error and adversarial loss (Goodfellow et al., 2014), the same architecture successfully extrapolates out-of-the-plane rotations of computer-generated faces. Furthermore, despite being trained end-to-end to predict only pixel-level information, our Predictive Generative Networks learn a representation of the latent structure of the underlying three-dimensional objects themselves. Importantly, we find that this representation is naturally tolerant to object transformations, and generalizes well to new tasks, such as classification of static images. Similar models trained solely with a reconstruction loss fail to generalize as effectively. We argue that prediction can serve as a powerful unsupervised loss for learning rich internal representations of high-level object features.

Motivation & Objective

• To investigate whether predictive video generation can serve as a powerful unsupervised learning framework for learning rich internal representations of visual structure.
• To evaluate whether models trained on future frame prediction learn disentangled, transformation-invariant features of underlying 3D objects.
• To compare the generalization performance of predictive models against reconstruction-based autoencoders in downstream classification tasks.
• To assess the impact of combining mean-squared error (MSE) and adversarial loss (AL) in improving prediction quality and representation learning.
• To test whether representations learned from dynamic stimuli generalize to static image recognition, especially under few-shot conditions.

Proposed method

• The model uses a CNN-LSTM-deCNN architecture (Encoder-Recurrent-Decoder) to predict future video frames from a sequence of input frames.
• It is trained end-to-end using a combination of mean-squared error (MSE) and adversarial loss (AL) to improve prediction realism and fidelity.
• The predictive loss encourages the network to learn an internal world model that captures temporal dynamics and structural invariances.
• Representations are extracted from the LSTM’s hidden state and evaluated on a static face identification task using SVMs.
• Control models are trained with reconstruction loss on static or dynamic frames, using autoencoder architectures with or without LSTMs.
• The model is evaluated on synthetic datasets: bouncing balls (physics-based) and rotating computer-generated faces (3D structure).

Experimental results

Research questions

• RQ1 Can a deep neural network trained solely on predicting future video frames learn disentangled representations of underlying 3D object structure?
• RQ2 How does predictive training compare to reconstruction-based training in learning transformation-tolerant features?
• RQ3 Does a predictive loss lead to better generalization to downstream tasks like static image classification?
• RQ4 What is the effect of combining MSE and adversarial loss on prediction quality and representation learning?
• RQ5 Can representations learned from dynamic video sequences generalize effectively to few-shot classification of static images?

Key findings

• The predictive generative network (PGN) achieves state-of-the-art performance on the standard 'bouncing balls' video prediction benchmark.
• The PGN with combined MSE and adversarial loss produces visually realistic and consistent predictions, especially for out-of-plane rotations of faces.
• The PGN trained with MSE alone achieves the highest classification accuracy (up to 94%) on a 50-way static face identification task, outperforming all reconstruction-based baselines.
• Even with fewer training examples, the predictive model generalizes significantly better than reconstruction-based models, especially in few-shot regimes.
• The representation learned by the PGN is inherently tolerant to object transformations, such as rotation, due to the inductive bias of temporal prediction.
• Models trained with a predictive loss generalize better than those trained with reconstruction loss, even when the latter are trained on the same data distribution.