[Paper Review] CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Wenyi Hong, Ming Ding+3 more|arXiv (Cornell University)|May 29, 2022

Human Pose and Action Recognition116 citations

TL;DR

CogVideo is a 9.4B-parameter transformer for open-domain text-to-video generation. It finetunes CogView2 with a multi-frame-rate hierarchical training strategy and dual-channel attention to achieve state-of-the-art results among public models.

ABSTRACT

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

Motivation & Objective

• Motivate text-to-video generation with large-scale pretrained transformers beyond text and text-to-image domains.
• Leverage pretrained text-to-image knowledge to facilitate video generation without full from-scratch pretraining.
• Propose a multi-frame-rate hierarchical training strategy to align text with temporal video semantics.
• Develop a dual-channel attention mechanism to integrate image-model knowledge into video generation efficiently.

Proposed method

• Finetune a pretrained CogView2 text-to-image model with a 9.4B parameter transformer for text-to-video generation.
• Introduce multi-frame-rate hierarchical training to align text prompts with frames by incorporating a frame-rate token and staged generation (sequential key frames then frame interpolation).
• Employ a dual-channel attention block that freezes CogView2 parameters and trains only the added temporal-spatial attention channels (attention-plus) to fuse image and video modeling.
• Apply Shifted Window (Swin) attention extended to autoregressive generation to enable partial parallelism and reduce memory cost.
• Use CogLM-style bidirectional/unidirectional masking to allow bidirectional context for interpolation while maintaining autoregressive frame generation.

Experimental results

Research questions

• RQ1 How can open-domain text-to-video generation achieve strong performance using existing text-to-image pretrained models?
• RQ2 Does a multi-frame-rate hierarchical training strategy improve text-video alignment and movement semantics?
• RQ3 Can a dual-channel attention mechanism effectively transfer image-language knowledge to video generation without full retraining?
• RQ4 What is the impact of using shifted window attention on autoregressive video generation efficiency?

Key findings

• CogVideo achieves state-of-the-art qualitative and quantitative performance among publicly available text-to-video models on UCF-101 and Kinetics-600.
• Hierarchical multi-frame-rate generation improves text-video alignment and motion semantics over single-stage generation.
• Finetuning CogView2 with dual-channel attention (attention-plus) yields better results than training from scratch or using random initialization, while keeping CogView2 parameters frozen.
• Shifted Window attention enables partial frame-level parallelism in autoregressive video generation, accelerating inference.
• Human evaluation shows CogVideo outperforms baselines across frame texture, motion realism, and semantic relevance.
• Ablation studies demonstrate the superiority of the hierarchical approach and CogView2-initialized models over random initialization.