[Paper Review] Adversarial Video Generation on Complex Datasets

Aidan Clark, Jeff Donahue+1 more|arXiv (Cornell University)|Jul 15, 2019

Generative Adversarial Networks and Image Synthesis69 references148 citations

TL;DR

DVD-GAN introduces a scalable dual-discriminator GAN for high-fidelity video generation on Kinetics-600, achieving state-of-the-art results for video synthesis and prediction.

ABSTRACT

Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600, as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

Motivation & Objective

• Aim to extend high-fidelity image generation success to the video domain using large-scale datasets.
• Develop a scalable GAN architecture capable of producing long, high-resolution videos.
• Establish strong baselines for class-conditional video synthesis on Kinetics-600.
• Evaluate on video synthesis and video prediction to benchmark temporal dynamics and quality.

Proposed method

• Build on BigGAN to create a Dual Video Discriminator GAN (DVD-GAN) for videos.
• Introduce two discriminators: a Spatial Discriminator (D_S) and a Temporal Discriminator (D_T).
• Downsample the input to D_T with a function phi to reduce computational load while preserving feedback.
• Sample k frames for D_S to judge per-frame content, summing their scores for final D_S output.
• Discriminator objective uses hinge loss with D_S and D_T supplying learning signals without full-video processing.
• Train on TPU pods with large-scale distributed training to handle 256×256 and up to 48-frame videos.

Experimental results

Research questions

• RQ1 Can a scalable GAN architecture generate high-fidelity, long-range videos on a diverse dataset like Kinetics-600?
• RQ2 Does decomposing discrimination into spatial and temporal components preserve feedback necessary for realism at high resolutions?
• RQ3 What are the effects of downsampling and frame sampling (k) on synthesis quality and diversity?
• RQ4 How does DVD-GAN perform on class-conditional video synthesis and future video prediction compared to prior methods?

Key findings

• DVD-GAN achieves state-of-the-art Inception Score on UCF-101 for video synthesis.
• On Kinetics-600, DVD-GAN attains high-fidelity samples at 64×64, 128×128, and 256×256 with up to 48 frames, demonstrating scalable performance.
• For synthesis on Kinetics-600, reported FID and IS improve over baselines across multiple resolutions and frame lengths.
• For prediction, DVD-GAN-FP achieves notably lower Fréchet Video Distance than prior adversarial models on Kinetics-600 and BAIR datasets.
• The dual-discriminator setup significantly reduces computational burden while maintaining strong feedback signals for realism across space and time.