[Paper Review] Stochastic Adversarial Video Prediction

Alex X. Lee, Richard Zhang+4 more|arXiv (Cornell University)|Apr 4, 2018

Adversarial Robustness in Machine LearningComputer Science49 references226 citations

TL;DR

The paper presents SAVP, a stochastic video prediction model that combines VAE-based latent variables with GAN-based adversarial training to produce diverse and realistic future video frames, outperforming prior methods in realism and diversity.

ABSTRACT

Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior and concurrent work in these aspects.

Motivation & Objective

• Address the multimodal nature of future video prediction by modeling stochasticity.
• Integrate latent variable modeling with adversarial training to improve realism and diversity.
• Evaluate how VAE and GAN components complement each other in stochastic video prediction.
• Compare SAVP to prior VAE-based and GAN-based methods across realism, diversity, and accuracy.
• Propose evaluation strategies that include human judgments and perceptual diversity metrics.

Proposed method

• We predict future frames using a recurrent generator that takes initial frames and time-varying latent codes.
• Training combines variational lower bound objectives with adversarial losses (VAE-GAN framework).
• Latent codes are inferred with an encoder to form a posterior, regularized toward a standard Gaussian prior.
• A separate video discriminator (and a VAEspecific discriminator) guides realism by matching joint video distributions.
• The generator is a convolutional LSTM with skip connections, conditioned on latent codes along the channel dimension.
• Evaluation uses both qualitative and quantitative metrics, including human judgments and perceptual diversity measures.

Experimental results

Research questions

• RQ1 Can a VAE-GAN architecture produce both diverse and realistic stochastic video predictions?
• RQ2 Does combining latent variable modeling with adversarial training outperform purely VAE or purely GAN approaches in video prediction?
• RQ3 How do realism, diversity, and accuracy trade off across SAVP variants on real-world datasets?
• RQ4 What evaluation strategies best reflect human judgments of video realism and predictive diversity?

Key findings

• The SAVP model improves realism over prior VAE-based methods and diversity over GAN-based methods.
• VAE-based variants yield higher diversity, while GAN-based variants yield higher realism; SAVP balances both.
• Standard pixel-wise metrics (PSNR/SSIM) may not align with human judgments, so human 2AFC tests were used.
• Diversity is measured via VGG-based perceptual distances, showing SAVP maintains diverse futures.
• On BAIR and KTH datasets, SAVP achieves higher realism in human evaluations than prior methods.
• Ablations show the necessity of both the VAE and GAN components for best overall performance.