[Paper Review] VideoGPT: Video Generation using VQ-VAE and Transformers
VideoGPT uses a VQ-VAE to compress videos into discrete latents and a GPT-style autoregressive transformer to model those latents, enabling competitive video generation with a simple, scalable pipeline.
We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html
Motivation & Objective
- Investigate whether a likelihood-based, autoregressive model can scale to natural video generation.
- Explore using a discretized latent space (VQ-VAE) to reduce spatio-temporal complexity.
- Assess the impact of axial attention and latent space design on video realism and fidelity.
- Demonstrate conditional and unconditional video generation capabilities.
- Provide ablations to guide reproducible, minimalistic transformer-based video generation.”],
- method (concise)Bullets interpreting as 3-6 bullets on the proposed method (key techniques/equations)
- ["Train a VQ-VAE with 3D convolutions and axial attention to learn downsampled discrete latents of videos.","Model the latent sequence autoregressively with a GPT-like transformer using spatio-temporal position encodings.","Use learned cross-attention or conditional norms for action/class conditioning of the prior.","Decode the latent samples back to full-resolution video via the VQ-VAE decoder.","Train with maximum likelihood and apply dropout for regularization in the prior."]
- research_questions 2-5 concrete research questions the paper investigates
- ["Can VideoGPT generate high-fidelity videos on real datasets like BAIR, UCF-101, and TGIF?","How do architectural choices (axial attention, latent size, codebook count, transformer depth) affect quality?","Is the approach competitive with state-of-the-art GANs for video generation?","What is the effect of conditioning mechanisms on conditional video generation?"]
- key_findings:["VideoGPT achieves FVD of 103.3 on BAIR (vs. 103.3 for TrIVD-GAN-FP) and 94±2 for Video Transformer, showing competitive quality with GAN-based methods.","Unconditional VideoGPT samples on UCF-101 achieve IS 24.69±0.30, competitive with several baselines and below DVD-GAN’s 32.97±1.7.","Ablations show axial attention improves reconstruction NMSE from 0.0041 to 0.0033 and FVD from 15.3 to 14.9 on BAIR (VQ-VAE).","Larger prior network capacity (up to 8-16 transformer layers) improves FVD and sample quality on BAIR.","Optimal latent configuration around 8×32×32 (space-time downsampling) yields best sample quality while maintaining reconstruction fidelity."," Using a single VQ-VAE codebook often yields best sample quality compared to multiple codebooks in their experiments."]
- table_headers:["Method","FVD (downarrow)"],
- table_rows:[ ["VideoGPT (ours)","103.3"], ["TrIVD-GAN-FP", "103.3"], ["Video Transformer", "94±2"], ["DVD-GAN-FP", "109.8"], ["SV2P", "262.5"], ["LVT", "125.8"], ["SAVP", "116.4"], ["IS on UCF-101: VideoGPT", "24.69±0.30"], ["IS on DVD-GAN", "32.97±1.7"], ["TGIF samples (uncond)", "-" ]]} }{
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.