[Paper Review] Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Tianfan Xue, Jiajun Wu+2 more|arXiv (Cornell University)|Jul 9, 2016

Advanced Vision and Imaging8 references145 citations

TL;DR

A probabilistic framework using a conditional variational autoencoder and a cross convolutional network to synthesize multiple plausible future frames from a single image without supervision, capturing motion-conditioned distributions.

ABSTRACT

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.

Motivation & Objective

• Motivate and model the conditional distribution of future frames given a single input image.
• Learn a content-aware, probabilistic motion representation without annotations.
• Enable sampling of diverse, realistic future frames that reflect intrinsic motion ambiguity.
• Demonstrate applicability to visual analogy-making and analyze learned representations.

Proposed method

• Introduce a conditional variational autoencoder to model p(v|I,z) and sample from p(z|v,I) to generate future frames J=I+v.
• Propose a cross convolutional layer that applies image-dependent motion kernels to multi-scale feature maps to synthesize a difference image v.
• Use an image encoder over a four-scale pyramid and a motion encoder to obtain a latent motion code z.
• Decoder combines learned motion kernels and feature maps to regress the Eulerian motion v.
• Train with a reconstruction objective on pairs of consecutive frames using KL-divergence regularization and reparameterization.
• Test time: sample z from a prior (empirical motion distribution) and generate multiple future frames J=I+v for a single input image I.

Experimental results

Research questions

• RQ1 Can a probabilistic model capture multiple plausible future frames conditioned on a single image?
• RQ2 Does a cross convolutional network that learns kernel-weighted motions for image regions better model Eulerian motion than prior approaches?
• RQ3 How well does the model generalize to synthetic and real-world video data without supervision?
• RQ4 Can the learned representations support tasks such as visual analogy-making and interpretation of motion channels?

Key findings

• The model learns a sparse, high-level motion representation z (with less than 30 PCA components explaining 95% of variance in z mean).
• Samples from the model closely approximate the ground-truth motion distributions on synthetic shape data, outperforming flow-transfer and non-VAE baselines.
• On sprites and real video datasets, the approach generates realistic, diverse future frames and scores higher in human judgment experiments than flow-based baselines.
• The framework enables zero-shot visual analogy-making by transferring learned motion relationships to new inputs, outperforming some supervised analogies methods.
• Feature maps learned by the network naturally detect objects and contours, indicating meaningful motion-aware representations.