[Paper Review] GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

Chenfei Wu, Lun Huang+6 more|arXiv (Cornell University)|Apr 30, 2021

Multimodal Machine Learning Applications31 references78 citations

TL;DR

GODIVA is a text-to-video pretrained model that generates open-domain videos from natural language descriptions using a VQ-VAE with a three-dimensional sparse attention mechanism, pretrained on Howto100M and evaluated with new Relative Matching metrics.

ABSTRACT

Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.

Motivation & Objective

• Address the challenge of generating semantically coherent videos from natural language descriptions.
• Propose a three-dimensional sparse attention mechanism to efficiently model temporal, row, and column relationships in video token generation.
• Introduce a VQ-VAE based framework to map videos to discrete tokens for autoregressive generation.
• Pretrain on a large-scale text-video dataset (Howto100M) and evaluate on real-world and synthetic benchmarks.
• Develop a relative metric (Relative Matching) to automatically assess video generation quality against text.

Proposed method

• Use a VQ-VAE auto-encoder to encode video frames into discrete tokens and reconstruct frames from those tokens.
• Model P(z|t) with an autoregressive generator that consumes text embeddings and discrete video tokens.
• Apply three-dimensional sparse attention with Temporal, Row, and Column components to reduce quadratic complexity to near-linear.
• Train with cross-entropy loss to predict the next visual token in a sequence.
• Pretrain on Howto100M and fine-tune on MSR-VTT; optionally apply CLIP-based ranking to improve RM scores.

Experimental results

Research questions

• RQ1 Can GODIVA generate coherent, open-domain videos from natural language descriptions?
• RQ2 Does a three-dimensional sparse attention mechanism enable efficient and scalable text-to-video generation?
• RQ3 How well does the model transfer to unseen text (zero-shot) and to downstream datasets (fine-tuning)?

Key findings

• GODIVA demonstrates the ability to generate semantically coherent videos from text and supports zero-shot generation with unseen descriptions.
• Three-dimensional sparse attention substantially reduces computation from quadratic to near-linear complexity while maintaining performance.
• A new Relative Matching (RM) metric correlates with semantic alignment and visual quality across experiments.
• Ablations show Row Attention as particularly important for performance; CLIP ranking further boosts RM.
• Pretraining on Howto100M enables effective fine-tuning on MSR-VTT and competitive zero-shot results.