[Paper Review] Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

William Lotter, Gabriel Kreiman+1 more|arXiv (Cornell University)|May 25, 2016

Advanced Vision and Imaging60 references418 citations

TL;DR

PredNet, a predictive coding-inspired deep recurrent CNN, learns to predict future video frames in an unsupervised manner and develops representations useful for decoding latent object parameters and downstream tasks like steering angle estimation.

ABSTRACT

While great strides have been made in using deep learning algorithms to solve supervised learning tasks, the problem of unsupervised learning - leveraging unlabeled examples to learn about the structure of a domain - remains a difficult unsolved challenge. Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world. We describe a predictive neural network ("PredNet") architecture that is inspired by the concept of "predictive coding" from the neuroscience literature. These networks learn to predict future frames in a video sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers. We show that these networks are able to robustly learn to predict the movement of synthetic (rendered) objects, and that in doing so, the networks learn internal representations that are useful for decoding latent object parameters (e.g. pose) that support object recognition with fewer training views. We also show that these networks can scale to complex natural image streams (car-mounted camera videos), capturing key aspects of both egocentric movement and the movement of objects in the visual scene, and the representation learned in this setting is useful for estimating the steering angle. Altogether, these results suggest that prediction represents a powerful framework for unsupervised learning, allowing for implicit learning of object and scene structure.

Motivation & Objective

• Motivate unsupervised learning from unlabeled video by predicting future frames.
• Develop a predictive coding-inspired architecture (PredNet) with local predictions and error-based communication.
• Demonstrate that prediction-learned representations support decoding latent factors (e.g., pose) and improve downstream tasks.
• Show scalability to natural video sequences (car-mounted cameras) and utility for estimating steering angles.

Proposed method

• Propose PredNet: a stacked, recurrent convolutional network with four components per layer: input A_l, representation R_l, prediction ĤA_l, and error E_l.
• Use ConvLSTM units for R_l and learn by minimizing a weighted sum of layer-wise prediction errors across time (L_train).
• Compute A_l from the bottom up (A_0 = x_t; A_l for l>0 via MaxPool(ReLU(Conv(E_{l-1}))); ĤA_l from R_l via Conv and ReLU; E_l as concatenation of positive/negative prediction errors (ReLU(A_l - ĤA_l) and ReLU(ĤA_l - A_l)).
• Train with Adam; explore two loss settings: PredNet_L0 (loss only at the lowest layer) and PredNet_Lall (loss at lowest and upper layers with smaller weights).
• Two-pass update scheme: top-down R_l states via ConvLSTM, then forward pass to compute predictions, errors, and higher-layer targets.

Experimental results

Research questions

• RQ1 Can predictive coding-inspired networks learn useful unsupervised representations from video by predicting future frames?
• RQ2 Do PredNet representations facilitate decoding of latent object parameters (e.g., pose, identity) and improve downstream tasks such as static object recognition?
• RQ3 Do PredNet models scale to naturalistic video (car-mounted camera) and capture ego-motion and object motion, enabling useful tasks like steering angle estimation?

Key findings

• PredNet outperforms baselines on rotating-face synthetic sequences in both MSE and SSIM (Rotating Faces: L0 MSE 0.0152, SSIM 0.937; Lall MSE 0.0157, SSIM 0.921; CNN-LSTM Enc.-Dec: MSE 0.0180, SSIM 0.907).
• On CalTech Pedestrian data, PredNet/L0 achieves MSE 3.13e-3 and SSIM 0.884; PredNet/Lall achieves MSE 3.33e-3 and SSIM 0.875; CNN-LSTM Enc.-Dec achieves MSE 3.67e-3 and SSIM 0.865; Copy Last Frame performs worst (MSE 7.95e-3, SSIM 0.762).
• Latent parameter decoding: representations from R_l improve linear decoding of latent factors (pan/roll velocity, pan angle, PC1) versus random networks; Lall particularly boosts first PC decoding.
• Static face classification with linear SVM shows PredNet representations outperform autoencoder and Ladder Network variants across training-set sizes, with Lall often yielding higher accuracy than L0.
• Steering angle estimation on Comma.ai data: with 1k labeled examples, linear readout on PredNet_L0 explains 74% variance in steering angle, outperforming the CNN-LSTM Enc.-Dec by ~35%; with 25k labels, PredNet_L0 MSE ~2.14 (deg^2).
• PredNet demonstrates robust frame prediction in natural scenes (KITTI) and reasonable generalization to CalTech Pedestrian test sequences; predicted frames can fill in occluded regions and handle camera motion.