[Paper Review] Movie Gen: A Cast of Media Foundation Models

Adam Polyak, Amit Zohar+86 more|arXiv (Cornell University)|Oct 17, 2024

Cinema and Media Studies7 citations

TL;DR

Movie Gen presents a cast of foundation models for high-quality 1080p HD video with synchronized audio, personalized videos, and precise video editing, trained with Flow Matching on large-scale image, video, and audio data, achieving state-of-the-art results across text-to-video, video personalization, video editing, and text-to-audio tasks.

ABSTRACT

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Motivation & Objective

• Motivate the development of a unified, scalable set of media generation foundation models (image, video, audio).
• Demonstrate state-of-the-art capabilities in text-to-video, video personalization, editing, and audio generation.
• Describe architectures, training recipes, data curation, and inference optimizations enabling long-context, high-resolution video generation.
• Introduce post-training procedures for personalization and precise editing to extend baseline generation capabilities.
• Provide benchmarks and publicly available resources to accelerate research in media generation models.

Proposed method

• Train a joint image and video foundation model (Movie Gen Video) on large-scale image and video data using a spatio-temporally compressed latent space (TAE).
• Use Flow Matching as the training objective to model velocity in latent space for video and image generation.
• Incorporate a 30B parameter Transformer backbone with full bi-directional attention and cross-attention conditioning from multiple text encoders.
• Add a separate 13B parameter Movie Gen Audio model for video-to-audio and text-to-audio generation with synchronized outputs.
• Implement post-training procedures for personalization (conditioning on an image) and precise editing (text-guided) without large supervised editing datasets.
• Employ spatial upsampling (HD 1080p) via a video-to-video upsampler and multi-diffusion techniques for temporal consistency.

Experimental results

Research questions

• RQ1 Can a single foundation model framework jointly generate high-quality images and videos with synchronized audio?
• RQ2 How can personalization and precise editing be achieved for videos without large supervised editing datasets?
• RQ3 What are the benefits of Flow Matching versus diffusion-based training for long-context video generation?
• RQ4 How does multi-modal conditioning (text prompts, reference images) affect generation quality and consistency?
• RQ5 What are the scalability and efficiency strategies (data, compute, parallelism) enabling 30B-parameter video models?

Key findings

• The Movie Gen Video 30B model can generate high-quality 16-second HD videos at 16 FPS with synchronized audio.
• Movie Gen Audio 13B model achieves state-of-the-art results in video audio generation including sound effects and music.
• Post-training personalization enables videos featuring specific persons while preserving identity.
• Post-training precise editing enables targeted edits to real or generated videos guided by text instructions.
• The approach achieves state-of-the-art performance on text-to-video generation and on video personalization and editing tasks, and audio generation benchmarks.
• Benchmarks for Movie Gen Video and Movie Gen Audio are publicly released to enable future evaluation.