[Paper Review] Make-A-Video: Text-to-Video Generation without Text-Video Data

Uriel Singer, Adam Polyak+11 more|arXiv (Cornell University)|Sep 29, 2022

Multimodal Machine Learning ApplicationsComputer Science311 citations

TL;DR

Make-A-Video converts a pretrained text-to-image model into a text-to-video system by adding spatiotemporal layers and a frame interpolation module, achieving state-of-the-art results without using paired text-video data. It leverages image priors and unlabeled video data for training.

ABSTRACT

We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

Motivation & Objective

• Leverage large-scale text-image priors to bootstrap video generation without paired text-video data.
• Extend 2D diffusion architectures to the temporal domain via pseudo-3D convolutions and attention.
• Incorporate frame interpolation and spatiotemporal super-resolution to produce high-resolution, high-frame-rate videos.
• Enable controllable frame rate and throughput through fps conditioning and temporal modules.
• Evaluate zero-shot and fine-tuned performance against existing text-to-video systems across standard benchmarks.

Proposed method

• Start from a pretrained text-to-image (T2I) model trained on text-image pairs (P, D, SR_l, SR_h).
• Extend spatial layers to the temporal domain with pseudo-3D (P3D) convolutions and attention, initialized to identity for smooth transition.
• Add a frame-interpolation network (↑F) and temporal conditioning (fps) to generate higher frame-rate videos.
• Train temporal layers on unlabeled video data after initial image-only training, using 16 frames per clip with randomized fps.
• Use a two-stage upscaling pipeline (SR_l^t, SR_h) across space and time to achieve higher spatial resolution and temporal coherence.
• Fine-tune a temporal decoder for masked frame interpolation to enable temporal upsampling.
• Inference follows a pipeline: prior P to image embedding, D^t to 16x64x64 frames, ↑F for interpolation, SR_l^t and SR_h for upscaling to 256x256 and 768x768 respectively, producing the final video.

Experimental results

Research questions

• RQ1 Can a text-to-image foundation model be effectively repurposed to generate video without relying on paired text-video data?
• RQ2 Do spatiotemporal extensions of diffusion blocks, initialized from 2D image priors, enable coherent motion and faithful text alignment in videos?
• RQ3 How do frame interpolation and spatiotemporal super-resolution affect video quality and frame-rate under text prompts?
• RQ4 How does zero-shot and fine-tuned T2V performance compare to existing state-of-the-art methods on standard benchmarks?
• RQ5 What is the impact of fps conditioning on video generation diversity and realism?

Key findings

• Make-A-Video achieves state-of-the-art results in text-to-video generation across qualitative and quantitative measures.
• Zero-shot MSR-VTT evaluation: Make-A-Video outperforms GODIVA and NÜWA and surpasses CogVideo in English and Chinese settings (Table 1).
• UCF-101 zero-shot: Make-A-Video achieves IS of 33.00 and FVD of 367.23, outperforming several prior methods; finetuning yields 82.55 IS and 81.25 FVD (Table 2).
• Human evaluation shows Make-A-Video is preferred for quality and faithfulness across DrawBench and the authors’ eval set, with strong margins over CogVideo and VDM baselines (Table 3).
• Frame interpolation (↑F) is preferred over FILM for motion realism in user studies (62% on one set, 54% on another).
• The approach uses open-source datasets (WebVid-10M, HD-VILA-100M) and does not rely on paired text-video data, enabling scalable training.