[Paper Review] Flexible Diffusion Modeling of Long Videos

William R. Harvey, Saeid Naderiparizi+3 more|arXiv (Cornell University)|May 23, 2022

Generative Adversarial Networks and Image SynthesisComputer Science102 citations

TL;DR

Introduces Flexible Diffusion Model (FDM), a DDPM-based framework that can sample any subset of video frames conditioned on any other subset, enabling long video generation/completion and dataset-specific optimization of sampling schemes; releases CARLA Town01 dataset.

ABSTRACT

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.

Motivation & Objective

• Address the challenge of coherent, photorealistic long-video generation beyond short sequences.
• Develop a DDPM-based model that can flexibly condition on and marginalize over arbitrary frame subsets at test time.
• Explore and optimize sampling schemes to balance computational constraints with video quality.
• Introduce a new autonomous driving video dataset (CARLA Town01) and semantics-based evaluation metrics for video modeling.

Proposed method

• Extend a DDPM-based image architecture with temporal attention and a novel relative frame-position encoding.
• Train a conditional DDPM with a meta-learning objective that supports conditioning on any number of past/future frames under a fixed compute budget K.
• Define a broad training task distribution u(X, Y) over latent and observed frame indices to enable learning of flexible conditioning across variable frame sets.
• Represent videos as 4-D tensors and use a 4-D U-Net with temporal and spatial attention to propagate information across frames.
• Implement training with training batch padding to efficiently handle variable-length conditioning within fixed batch sizes.
• Provide multiple test-time sampling schemes (Autoregressive, Long-range, Hierarchy variants) and an optimization procedure to select conditioning frames that minimize the diffusion loss.

Experimental results

Research questions

• RQ1 How can a diffusion-based video model be conditioned on arbitrary subsets of frames at test time?
• RQ2 Can a single model be trained to handle diverse conditioning tasks and still deliver high-quality long videos under computational constraints?
• RQ3 What sampling schemes yield the best trade-off between video fidelity and coherence for different datasets?
• RQ4 Does optimizing conditioning strategies at test time improve quantitative video-modeling metrics compared to fixed schemes?
• RQ5 What semantics-driven metrics can evaluate long videos generated in driving simulation environments?

Key findings

• FDM enables sampling of long videos (up to 25 minutes) conditioned on arbitrary subsets of frames across datasets.
• Various sampling schemes (Autoregressive, Long-range, Hierarchy variants) show dataset-dependent performance, with hierarchical schemes often improving Fréchet Video Distance (FVD) on some tasks while autoregressive schemes excel on others.
• Optimizing conditioning frames with offline scheme optimization (Opt. autoreg, Opt. hierarchy-2) improves FVD and related metrics on several datasets.
• Compared to baselines (CWVAE, TATS, VDM), FDM achieves lower FVDs or higher accuracies depending on dataset and scheme, and offers test-time flexibility to explore schemes without retraining.
• A new CARLA Town01 video dataset is released with semantics-oriented evaluation (outlier percentage, Wasserstein distance) to assess long-video realism and semantic coherence.