[Paper Review] VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

M. N. Anil Kumar, Mohammad Babaeizadeh+5 more|arXiv (Cornell University)|Mar 4, 2019

Video Analysis and SummarizationComputer Science57 references76 citations

TL;DR

VideoFlow extends flow-based generative models to conditional video prediction, enabling exact likelihood optimization, diverse stochastic futures, and faster frame synthesis than autoregressive video models.

ABSTRACT

Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modelling of video.

Motivation & Objective

• Motivate stochastic video prediction in which multiple futures are possible.
• Propose a flow-based model for conditioning on past frames to synthesize future frames.
• Introduce a latent dynamical system to model temporal evolution of the flow’s latent state.
• Enable exact log-likelihood evaluation for video generation to avoid adversarial training artifacts.

Proposed method

• Use a multi-scale invertible flow to map frames x_t to latent codes z_t with per-frame latent variables z_t^(l).
• Model an autoregressive prior p(z) across time over all z_t^(l) to capture temporal dynamics.
• Train by maximizing exact log-likelihood combining flow Jacobian terms and the autoregressive latent prior.
• Condition the flow-based generator on past frames while keeping the latent dynamics autoregressive in time.
• Leverage 2-D convolutions with autoregressive priors to avoid temporal artifacts and enable longer sequences.
• Optionally adjust sampling temperature to trade off diversity and realism.

Experimental results

Research questions

• RQ1 Can a conditional flow-based model produce high-quality stochastic video predictions with exact likelihood optimization?
• RQ2 How does VideoFlow compare to VAE- and autoregressive-based video prediction methods in terms of realism, diversity, and sampling speed?
• RQ3 Does an autoregressive latent dynamics prior enable coherent multi-frame video generation without expensive 3-D convolutions?
• RQ4 Can the model generate longer horizon predictions while maintaining temporal consistency under occlusions?

Key findings

• VideoFlow yields competitive stochastic video prediction results on BAIR, approaching state-of-the-art VAE-based models.
• On the Stochastic Movement Dataset, VideoFlow achieved a higher real-vs-fake fooling rate (31.8%) than SAVP-VAE (16.4%) and SV2P (17.5%).
• VideoFlow achieves faster test-time synthesis than pixel-level autoregressive models (e.g., 64x64x20 frames in under 3.5 seconds on a NVIDIA P100).
• The model directly optimizes data likelihood, avoiding adversarial training artifacts and enabling direct evaluation via log-likelihoods.
• VideoFlow attains better Bits-Per-Pixel on BAIR action-free data (1.87) than several baselines, indicating stronger likelihood-based modeling.
• Latent-space interpolations on BAIR reveal temporally cohesive motion, with different levels capturing motions at different scales.