[Paper Review] Imagen Video: High Definition Video Generation with Diffusion Models

Jonathan Ho, William Chan+9 more|arXiv (Cornell University)|Oct 5, 2022

Generative Adversarial Networks and Image SynthesisComputer Science346 citations

TL;DR

Imagen Video uses a cascade of diffusion models to generate high-definition, text-conditioned videos with spatial and temporal super-resolution, achieving fast, high-quality sampling through progressive distillation and classifier-free guidance.

ABSTRACT

We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models. We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models. In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting. Finally, we apply progressive distillation to our video models with classifier-free guidance for fast, high quality sampling. We find Imagen Video not only capable of generating videos of high fidelity, but also having a high degree of controllability and world knowledge, including the ability to generate diverse videos and text animations in various artistic styles and with 3D object understanding. See https://imagen.research.google/video/ for samples.

Motivation & Objective

• Demonstrate a cascaded diffusion approach for high-definition text-to-video generation.
• Show how text conditioning and frozen text encoders transfer from image to video generation.
• Investigate diffusion parameterizations, conditioning augmentation, and progressive distillation for video sampling efficiency.
• Assess the model's controllability, 3D understanding, and diverse artistic styles in video generation.

Proposed method

• Adopt a cascade of seven diffusion sub-models (base video model plus three spatial SR and three temporal SR models) to produce 1280x768 videos at 24 fps.
• Use a frozen T5-XXL text encoder to condition all diffusion models on the input text.
• Employ v-prediction parameterization for all models to improve stability and reduce color artifacts.
• Train all models jointly on image-text and video-text data, with images treated as single-frame videos to enable larger datasets.
• Apply classifier-free guidance to improve alignment with text prompts, including techniques like dynamic clipping and oscillating guidance to manage artifacts.
• Utilize progressive distillation to reduce sampling steps and accelerate generation, enabling a distilled pipeline with substantially faster sampling.

Experimental results

Research questions

• RQ1 How well can cascaded diffusion models generate high-definition, temporally coherent videos from text prompts?
• RQ2 What is the impact of parameterizations (v-prediction vs epsilon-prediction) on video sample quality and stability at high resolution?
• RQ3 Does joint training on images and videos improve video quality and enable style and 3D understanding transfer?
• RQ4 How effective is progressive distillation in delivering fast, high-quality video samples with guidance?

Key findings

• The model generates high-fidelity 1280x768 videos at 24 fps (5.3 seconds, 128 frames) with strong temporal coherence and alignment to prompts.
• Scaling the base video model consistently improves FVD and CLIP scores, indicating benefits from larger parameter counts.
• v-prediction outperforms epsilon-prediction at high resolutions, reducing color shifts and accelerating convergence.
• Distilled cascades achieve about 18x faster sampling with similar perceptual quality and ~36x fewer FLOPs compared to the original cascade.
• Joint image-video training enhances video quality and enables transfer of styles and 3D-like understanding.
• Classifier-free guidance substantially improves prompt alignment and sample quality; oscillating guidance mitigates saturation artifacts during sampling.