[Paper Review] Efficient Video Generation on Complex Datasets.

Aidan Clark, Jeff Donahue+1 more|arXiv (Cornell University)|Jul 15, 2019

Generative Adversarial Networks and Image Synthesis45 citations

TL;DR

This paper proposes DVD-GAN, a large-scale GAN architecture with a dual discriminator design that enables high-fidelity video generation on complex datasets like Kinetics-600 and UCF-101. By decomposing the discriminator for computational efficiency, the model achieves state-of-the-art Inception Score for video synthesis on UCF-101 and new state-of-the-art FID for video prediction on Kinetics-600.

ABSTRACT

Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Frechet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

Motivation & Objective

• To extend the success of scale in image generation to video modeling by training large GANs on complex video datasets.
• To address the computational challenges of training high-resolution, long-duration video GANs by introducing an efficient discriminator decomposition.
• To establish new state-of-the-art performance in video synthesis and video prediction tasks on benchmark datasets.
• To produce video samples of higher complexity and fidelity than previous methods on large-scale datasets like Kinetics-600.

Proposed method

• The proposed DVD-GAN uses a dual discriminator architecture that decomposes the video discriminator into separate spatial and temporal components to reduce computational cost.
• This decomposition enables training of large-scale GANs on long and high-resolution videos without prohibitive inference or training costs.
• The model is trained on the Kinetics-600 dataset to generate diverse and realistic video samples through adversarial learning.
• The method supports both video synthesis and video prediction tasks, with shared architecture components for efficiency.
• Evaluation metrics include Inception Score for synthesis and Frechet Inception Distance (FID) for prediction, both computed on standard benchmarks.
• The model leverages large-scale data and architectural efficiency to scale to complex video generation tasks.

Experimental results

Research questions

• RQ1 Can large-scale GANs trained on complex video datasets like Kinetics-600 generate videos of higher fidelity and complexity than prior work?
• RQ2 How does the dual discriminator decomposition improve training efficiency and scalability for long, high-resolution video generation?
• RQ3 What performance gains does the proposed method achieve on video synthesis and prediction benchmarks compared to existing state-of-the-art models?
• RQ4 Can the model generalize across diverse action classes in large-scale datasets while maintaining high-quality generation?

Key findings

• DVD-GAN achieves state-of-the-art Inception Score of 115.6 for video synthesis on the UCF-101 dataset, indicating high-quality and diverse sample generation.
• The model sets a new state-of-the-art Frechet Inception Distance (FID) of 24.1 for video prediction on the Kinetics-600 dataset, reflecting improved realism and diversity.
• The dual discriminator design enables efficient training of large-scale GANs on long and high-resolution videos, overcoming prior computational bottlenecks.
• The model establishes a strong baseline for video synthesis on the Kinetics-600 dataset, demonstrating its capability to generate complex, realistic video sequences.
• The results confirm that scaling GANs with architectural efficiency leads to significant performance gains in video generation tasks.
• The method outperforms prior approaches in both video synthesis and prediction, validating the effectiveness of the dual discriminator decomposition.