[Paper Review] Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure

Paul Henderson, Christoph H. Lampert+1 more|arXiv (Cornell University)|Jun 16, 2021

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes ViPL4S, a novel unsupervised video prediction model that reconstructs 3D dynamic scene structure from a single frame to generate temporally coherent, diverse video predictions. By estimating 4D scene structure—including 3D geometry, motion segmentation, and camera/object dynamics—using end-to-end training with only RGB video, it achieves state-of-the-art performance on real-world datasets, outperforming prior methods in both qualitative and quantitative metrics.

ABSTRACT

Our goal in this work is to generate realistic videos given just one initial frame as input. Existing unsupervised approaches to this task do not consider the fact that a video typically shows a 3D environment, and that this should remain coherent from frame to frame even as the camera and objects move. We address this by developing a model that first estimates the latent 3D structure of the scene, including the segmentation of any moving objects. It then predicts future frames by simulating the object and camera dynamics, and rendering the resulting views. Importantly, it is trained end-to-end using only the unsupervised objective of predicting future frames, without any 3D information nor segmentation annotations. Experiments on two challenging datasets of natural videos show that our model can estimate 3D structure and motion segmentation from a single frame, and hence generate plausible and varied predictions.

Motivation & Objective

• To address the challenge of generating temporally coherent and diverse video predictions from a single image without supervision.
• To leverage the prior knowledge that real-world videos depict a 3D environment with stable geometry and motion coherence, which existing unsupervised models fail to exploit.
• To learn 3D scene structure and motion segmentation from monocular RGB video alone, without any 3D annotations or supervision.
• To develop a conditional generative model that ensures future frames are rendered from a consistent 4D scene representation, improving temporal coherence.

Proposed method

• The model uses a variational autoencoder (VAE) framework with a latent Gaussian variable z to model uncertainty in future frames, conditioned on a single input frame x₀.
• It introduces a novel 4D scene representation F that factors geometry, texture, motion segmentation (ωⱼ), camera motion (Λₜ), and object transformations (Tₜʲ) into a disentangled, low-dimensional latent space.
• The 4D scene representation is learned end-to-end via reconstruction loss, with the model predicting future frames by rendering the scene at time t using differentiable neural rendering.
• Motion components are represented via a factorized transformation per object, enabling efficient modeling of coherently moving regions without dense optical flow.
• The model is trained using only pixel-level reconstruction loss and a VAE objective, with no 3D supervision, segmentation labels, or explicit geometric priors.
• A differentiable rendering pipeline computes pixel intensities by tracing rays through the 3D scene, enabling gradient-based optimization of scene geometry and motion.

Experimental results

Research questions

• RQ1 Can a model learn to infer 3D dynamic scene structure—including moving object segmentation and camera motion—from a single RGB frame without any 3D supervision?
• RQ2 Can an unsupervised video prediction model achieve better temporal coherence and diversity by explicitly modeling a 4D scene representation?
• RQ3 Does incorporating 3D scene structure as a disentangled latent factor improve video prediction quality compared to direct frame-to-frame generation?
• RQ4 How well can such a model generalize to real-world videos with complex camera motion and dynamic objects, without annotated 3D data?

Key findings

• The model achieves significantly better quantitative performance than the state-of-the-art method SRVP on both the Waymo and re-10k datasets, with lower FVD and KVD scores.
• On the Waymo dataset, ViPL4S achieves a Fréchet Video Distance (FVD) of 112.4, compared to 143.6 for SRVP, indicating a closer distribution to real video data.
• The model successfully learns to segment moving objects such as cars and pedestrians from a single frame, even without supervision, as shown in qualitative examples.
• The 6-component variant of ViPL4S underperforms the 4-component version, suggesting optimization challenges and potential overfitting when component count exceeds optimal capacity.
• The specialized vehicle-motion variant (ViPL4S veh) outperforms the general motion model, indicating that inductive biases improve training efficiency and performance.
• Despite limited supervision, the model generates plausible video clips with coherent camera motion and relative object motion, avoiding artifacts like object deformation or flickering seen in baseline models.