[Paper Review] Towards Accurate Generative Models of Video: A New Metric & Challenges

Thomas Unterthiner, Sjoerd van Steenkiste+4 more|arXiv (Cornell University)|Dec 3, 2018

Generative Adversarial Networks and Image SynthesisComputer Science50 references194 citations

TL;DR

This paper introduces Fréchet Video Distance (FVD), a new metric for evaluating video generative models by measuring the distributional distance between real and generated video features, incorporating both frame quality and temporal coherence. FVD outperforms traditional metrics like PSNR and SSIM in aligning with human judgment and is validated on a new benchmark, StarCraft 2 Videos (SCV), which exposes limitations in current models' long-term memory and relational reasoning capabilities.

ABSTRACT

Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent attempts at formulating generative models of video have had some success, current progress is hampered by (1) the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples, and (2) the wide gap between purely synthetic video data sets and challenging real-world data sets in terms of complexity. To this extent we propose Fr\'{e}chet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the current capabilities of generative models of video. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos, and provide initial benchmark results on SCV.

Motivation & Objective

• To address the lack of comprehensive metrics that evaluate visual quality, temporal coherence, and sample diversity in video generation.
• To develop a metric that operates on full video distributions rather than frame-level comparisons.
• To introduce a benchmark dataset, StarCraft 2 Videos (SCV), that tests long-term memory and relational reasoning in video generation.
• To validate the correlation of FVD with human judgment through a large-scale human study.
• To provide a comprehensive evaluation of state-of-the-art models across multiple datasets using FVD.

Proposed method

• Proposes Fréchet Video Distance (FVD) as a video extension of Fréchet Inception Distance (FID), using a 3D CNN to extract spatiotemporal features from video clips.
• Computes the Fréchet distance between the multivariate Gaussian distributions of real and generated video features, capturing both frame-level quality and temporal dynamics.
• Employs a pre-trained Inflated 3D ConvNet (I3D) as the feature extractor to encode video clips into high-dimensional embeddings.
• Validates FVD sensitivity to temporal and frame-level perturbations by adding noise to real videos and measuring FVD changes.
• Conducts a large-scale human study with 20,000 video samples across multiple models and datasets to correlate FVD with human perception.
• Introduces StarCraft 2 Videos (SCV), a scalable benchmark of 4 custom StarCraft 2 scenarios requiring relational reasoning and long-horizon memory.

Experimental results

Research questions

• RQ1 How well does FVD correlate with human judgment of video quality compared to PSNR and SSIM?
• RQ2 Can FVD detect both frame-level and temporal artifacts in generated videos?
• RQ3 To what extent do current state-of-the-art video generation models fail on complex, long-horizon video generation tasks?
• RQ4 How do different video generation models perform across diverse benchmarks including BAIR, KTH, and SCV?
• RQ5 Can FVD serve as a reliable, objective metric for unconditional video generation where ground truth is unavailable?

Key findings

• FVD shows a statistically significant negative correlation with human judgment (r = -0.640, τ = -0.189), outperforming SSIM and PSNR in alignment with human perception.
• FVD is sensitive to both frame-level and temporal perturbations, as demonstrated by noise injection experiments.
• On the SCV benchmark, no model successfully generated accurate long-term sequences for the RTwM scenario, indicating unresolved challenges in long-horizon generation.
• The CMS scenario caused most models to fail in modeling the correct disappearance sequence of mineral shards, despite learning the general disappearance pattern.
• On BAIR and KTH, models like SVP-FP and SAVP achieved the lowest FVD scores, indicating superior performance in generating temporally coherent and visually accurate videos.
• The study evaluated over 3,000 models across multiple datasets, consuming more than 100 GPU-years of computation, establishing a comprehensive benchmark for video generation.