[Paper Review] Diverse Video Generation from a Single Video

Niv Haim, Ben Feinstein+5 more|arXiv (Cornell University)|May 11, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes VGPNN, a non-parametric, nearest-neighbor-based method for diverse video generation from a single input video, leveraging space-time patch matching via PatchMatch for fast, high-resolution synthesis. It outperforms single-video GANs in visual quality and speed—reducing training time from days to seconds—while enabling novel applications like video analogies and spatio-temporal retargeting.

ABSTRACT

GANs are able to perform generation and manipulation tasks, trained on a single video. However, these single video GANs require unreasonable amount of time to train on a single video, rendering them almost impractical. In this paper we question the necessity of a GAN for generation from a single video, and introduce a non-parametric baseline for a variety of generation and manipulation tasks. We revive classical space-time patches-nearest-neighbors approaches and adapt them to a scalable unconditional generative model, without any learning. This simple baseline surprisingly outperforms single-video GANs in visual quality and realism (confirmed by quantitative and qualitative evaluations), and is disproportionately faster (runtime reduced from several days to seconds). Our approach is easily scaled to Full-HD videos. We also use the same framework to demonstrate video analogies and spatio-temporal retargeting. These observations show that classical approaches significantly outperform heavy deep learning machinery for these tasks. This sets a new baseline for single-video generation and manipulation tasks, and no less important -- makes diverse generation from a single video practically possible for the first time.

Motivation & Objective

• Address the impractical training time and poor visual quality of existing single-video GANs for video generation.
• Revive and scale classical patch-based video generation methods to enable high-resolution, diverse video synthesis from a single input.
• Establish a practical, non-learning baseline for video generation and manipulation tasks that outperforms deep learning-based alternatives.
• Extend the framework to new applications such as video analogies, sketch-to-video, and spatio-temporal retargeting.
• Demonstrate that simple, classical methods can surpass complex deep learning models in specific video generation tasks.

Proposed method

• Construct a spatio-temporal pyramid of the input video at multiple resolutions for coarse-to-fine processing.
• Apply a Video-Patch-Nearest-Neighbor (VPNN) module at each pyramid level to transfer spatial and temporal features from the input to the output.
• Use the PatchMatch algorithm for fast approximate nearest neighbor search in 3D space-time patches, enabling scalability.
• Inject stochastic noise at the coarsest pyramid level to induce diversity in generated outputs.
• Employ optical flow-based descriptors (e.g., RAFT) to extract dynamic structure for video analogies and layout-guided generation.
• Combine content video dynamics with style video appearance and motion via feature concatenation in the pyramid layers.

Experimental results

Research questions

• RQ1 Can a non-parametric, learning-free approach outperform deep learning-based single-video GANs in video generation quality and speed?
• RQ2 Is it feasible to generate diverse, high-resolution videos (e.g., Full-HD) from a single input video using classical patch-based methods?
• RQ3 Can the same framework be extended to tasks beyond unconditional generation, such as video analogies and spatio-temporal retargeting?
• RQ4 How does the use of PatchMatch for approximate nearest neighbor search impact runtime and memory efficiency in video generation?
• RQ5 To what extent can classical space-time patch matching achieve visual realism and coherence comparable to modern deep generative models?

Key findings

• VGPNN reduces inference time from 8 days (HP-VAE-GAN) to just 18 seconds for a single video generation, a 30,000x speedup.
• The method achieves superior visual quality, with a SVFID score of 0.0072 compared to 0.0081 for HP-VAE-GAN, indicating better patch distribution similarity.
• In user studies, 67.84% of participants preferred VGPNN-generated videos over HP-VAE-GAN in terms of sharpness, naturalness, and coherence.
• VGPNN generates full-HD (1280×1920) videos, whereas HP-VAE-GAN and SinGAN-GIF are limited to low-resolution outputs (e.g., 144×256).
• The framework enables diverse, unconditional video generation, video analogies, sketch-to-video translation, and conditional video inpainting.
• The approach outperforms single-video GANs in both quantitative metrics and qualitative perception, establishing a new practical baseline for single-video video generation.