[Paper Review] Stochastic Latent Residual Video Prediction

Jean-Yves Franceschi, Edouard Delasalles+3 more|arXiv (Cornell University)|Feb 21, 2020

Generative Adversarial Networks and Image Synthesis82 references40 citations

TL;DR

The paper introduces a fully latent stochastic video prediction model with a residual latent dynamics update, enabling non-autoregressive forecasting and higher frame-rate flexibility, and shows state-of-the-art results on multiple benchmarks.

ABSTRACT

Designing video prediction models that account for the inherent uncertainty of the future is challenging. Most works in the literature are based on stochastic image-autoregressive recurrent networks, which raises several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and temporal dynamics. However, no such model for stochastic video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model whose dynamics are governed in a latent space by a residual update rule. This first-order scheme is motivated by discretization schemes of differential equations. It naturally models video dynamics as it allows our simpler, more interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

Motivation & Objective

• Motivate learning predictive video models that capture future uncertainty in a self-supervised manner.
• Propose a fully latent, non-autoregressive temporal model with a stochastic residual update rule.
• Separate dynamic latent state evolution from frame synthesis to improve interpretability and efficiency.
• Incorporate a content variable to capture static scene information and aid frame generation.
• Demonstrate improved performance over baselines on standard stochastic video prediction benchmarks.

Proposed method

• Model frames as generated from latent states y_t with stochastic residual updates y_{t+1}=y_t+f_theta(y_t,z_{t+1}).
• Introduce a latent stochastic variable z_{t+1} ~ N(mu_theta(y_t), sigma_theta(y_t)) to drive dynamics.
• Use a content variable w, derived from conditioning frames, to represent static scene information and feed it to the frame decoder.
• Apply variational inference with an ELBO that includes a KL term for y_1 and z_t, plus log-likelihood terms for x_t given y_t and w.
• Allow a controllable frame-rate via Delta t, enabling generation at different frame rates without retraining.
• Train with a residual regularization term on f_theta to stabilize dynamics.
• Decode frames x_t from y_t and w using a CNN-based generator g_theta.

Experimental results

Research questions

• RQ1 Can a fully latent stochastic model with residual dynamics outperform autoregressive or traditional SSM-based approaches in stochastic video prediction?
• RQ2 Does separating content from dynamics improve learning efficiency and prediction quality?
• RQ3 Is the residual dynamics formulation compatible with generating videos at higher frame rates than used in training?
• RQ4 How does the proposed model compare to state-of-the-art baselines on standard stochastic video prediction benchmarks?
• RQ5 Can the model robustly predict diverse futures across datasets like Moving MNIST, KTH, Human3.6M, and BAIR?

Key findings

• Outperforms state-of-the-art baselines on several stochastic video prediction benchmarks (KTH, Human3.6M, BAIR).
• Demonstrates better long-horizon dynamic modeling than SVG and competitive variants (MLP/GRU) of the same residual framework.
• Shows ability to generate videos at higher frame rates by halving Delta t without retraining, maintaining performance.
• Disentangles dynamic content (y) from static content (w), enabling the model to focus dynamics in latent space.
• Residual dynamics with stochastic latent variables z_t provides advantages over purely deterministic or autoregressive approaches.
• FVD scores indicate strong performance across datasets, with notable gains on KTH and Human3.6M.