[Paper Review] Open-Sora: Democratizing Efficient Video Production for All

Z. Zheng, Xiangyu Peng+7 more|arXiv (Cornell University)|Dec 29, 2024

Urban Planning and Valuation10 citations

TL;DR

Open-Sora is an open-source diffusion-based video generation model supporting text-to-image, text-to-video, and image-to-video tasks, achieving up to 16 seconds at 720p with a Spatial-Temporal Diffusion Transformer and a compact 3D autoencoder.

ABSTRACT

Vision and language are the two foundational senses for humans, and they build up our cognitive ability and intelligence. While significant breakthroughs have been made in AI language ability, artificial visual intelligence, especially the ability to generate and simulate the world we see, is far lagging behind. To facilitate the development and accessibility of artificial visual intelligence, we created Open-Sora, an open-source video generation model designed to produce high-fidelity video content. Open-Sora supports a wide spectrum of visual generation tasks, including text-to-image generation, text-to-video generation, and image-to-video generation. The model leverages advanced deep learning architectures and training/inference techniques to enable flexible video synthesis, which could generate video content of up to 15 seconds, up to 720p resolution, and arbitrary aspect ratios. Specifically, we introduce Spatial-Temporal Diffusion Transformer (STDiT), an efficient diffusion framework for videos that decouples spatial and temporal attention. We also introduce a highly compressive 3D autoencoder to make representations compact and further accelerate training with an ad hoc training strategy. Through this initiative, we aim to foster innovation, creativity, and inclusivity within the community of AI content creation. By embracing the open-source principle, Open-Sora democratizes full access to all the training/inference/data preparation codes as well as model weights. All resources are publicly available at: https://github.com/hpcaitech/Open-Sora.

Motivation & Objective

• Democratize access to high-quality video generation by providing training code, data processing pipelines, and model weights.
• Reproduce and extend Sora-inspired techniques to enable controllable video synthesis.
• Develop efficient architectures (STDiT) and compression strategies to reduce training/inference costs.
• Support multi-resolution and multi-aspect-ratio video generation to maximize flexibility for downstream tasks.

Proposed method

• Introduce Spatial-Temporal Diffusion Transformer (STDiT) that decouples spatial and temporal attention.
• Use a 3D autoencoder to compress video data (4x temporal compression) initialized from a 2D VAE to maintain temporal fidelity.
• Initialize from a pre-trained image generation model (PixArt-α) and add temporal attention with zero-initialized projections.
• Apply RoPE for temporal attention and QK-normalization to improve stability in half-precision training.
• Employ a masking-based conditioning strategy for image/video conditioning and append conditioning scores (aesthetic, motion, camera motion) to captions.
• Adopt a bucket-based multi-resolution training scheme to handle diverse resolutions and aspect ratios efficiently.
• Utilize a multi-stage training curriculum across progressively higher-quality datasets and resolutions to balance cost and performance.

Experimental results

Research questions

• RQ1 Can Open-Sora reproduce and extend open-source diffusion-based video generation techniques to produce high-quality videos up to 16 seconds at 720p?
• RQ2 What architectural and training strategies best balance quality, speed, and resource usage for open-source video diffusion models?
• RQ3 How can conditioning signals (aesthetic, motion, camera motion) improve controllability in text-to-video generation?
• RQ4 What are effective data preprocessing and multi-resolution strategies to enable flexible video generation across resolutions and aspect ratios?

Key findings

• Open-Sora achieves state-of-the-art open-source video generation performance on VBench, surpassing several baselines in total, quality, and semantic scores.
• Version 1.2 of Open-Sora attains a total VBench score of 79.76, with 81.35 in quality and 73.39 in semantic scores.
• Compared models on VBench include CogVideo (67.01 total), Latte (77.29 total), LaVie (77.08 total), Show-1 (78.93 total), and multiple Open-Sora plan variants.
• A bucket-based multi-resolution training strategy effectively balances computational efficiency with diverse video formats.
• A 4x temporal compression in the 3D autoencoder preserves FPS and reduces training/inference costs while maintaining quality.