[Paper Review] Temporally Consistent Transformers for Video Generation

Wilson Yan, Danijar Hafner+2 more|arXiv (Cornell University)|Oct 5, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces TECO, a Temporally Consistent Transformer for video generation that achieves long-horizon temporal consistency by compressing video frames into discrete latents, applying a lightweight temporal transformer on the compressed sequence, and reconstructing high-fidelity videos via a spatial MaskGit. TECO outperforms prior models in temporal consistency and sampling speed on challenging long-range video benchmarks derived from 3D environments.

ABSTRACT

To generate accurate videos, algorithms have to understand the spatial and temporal dependencies in the world. Current algorithms enable accurate predictions over short horizons but tend to suffer from temporal inconsistencies. When generated content goes out of view and is later revisited, the model invents different content instead. Despite this severe limitation, no established benchmarks on complex data exist for rigorously evaluating video generation with long temporal dependencies. In this paper, we curate 3 challenging video datasets with long-range dependencies by rendering walks through 3D scenes of procedural mazes, Minecraft worlds, and indoor scans. We perform a comprehensive evaluation of current models and observe their limitations in temporal consistency. Moreover, we introduce the Temporally Consistent Transformer (TECO), a generative model that substantially improves long-term consistency while also reducing sampling time. By compressing its input sequence into fewer embeddings, applying a temporal transformer, and expanding back using a spatial MaskGit, TECO outperforms existing models across many metrics. Videos are available on the website: https://wilson1yan.github.io/teco

Motivation & Objective

• Address the lack of rigorous benchmarks for evaluating long-term temporal consistency in video generation models.
• Identify and quantify the failure modes of existing video generation models in maintaining consistency when revisiting previously observed scene regions.
• Propose a novel, efficient architecture that scales to long video sequences while preserving temporal coherence.
• Enable fast sampling and high-fidelity generation by decoupling spatial and temporal modeling through hierarchical compression and reconstruction.

Proposed method

• Curate three new video datasets—DMLab, Minecraft, and Habitat—by rendering camera walks through 3D procedural scenes to create long-range temporal dependencies.
• Use a VQ-GAN-based encoder to compress input video frames into discrete latent codes, reducing sequence length and computational cost.
• Apply a lightweight temporal transformer on the compressed latent sequence to model long-range dependencies efficiently, avoiding quadratic attention complexity.
• Reconstruct high-fidelity video frames using a spatial MaskGit applied independently per frame, enabling autoregressive refinement of spatial details.
• Design a two-stage generation process: first predict compressed latents via the temporal transformer, then reconstruct pixel-level images via MaskGit.
• Optimize training using cosine learning rate schedules, Adam optimizer, and fixed total training steps across all models and datasets.

Experimental results

Research questions

• RQ1 How do state-of-the-art video generation models perform in maintaining temporal consistency when revisiting previously seen locations in long video sequences?
• RQ2 Can a transformer-based model effectively model long-range dependencies in video generation when applied to compressed latent representations?
• RQ3 To what extent does hierarchical compression and reconstruction improve sampling speed and temporal consistency compared to end-to-end autoregressive models?
• RQ4 How do different architectural choices (e.g., downsampling factor, model depth) affect performance and efficiency on long-horizon video generation tasks?
• RQ5 Can a unified framework achieve both high-fidelity generation and fast inference while maintaining consistency over hundreds of frames?

Key findings

• TECO achieves state-of-the-art performance on long-horizon video prediction, generating temporally consistent videos over 264 frames with low LPIPS (0.15–0.25) on DMLab and Minecraft.
• TECO reduces sampling time by orders of magnitude compared to prior models, enabling fast inference while maintaining high perceptual quality.
• The model maintains consistent scene content when revisiting previously observed regions, unlike prior models that generate inconsistent or novel content.
• On the DMLab and Minecraft datasets, TECO outperforms strong baselines such as FitVid and CW-VAE in both FVD and LPIPS metrics, with FVD improvements of up to 15%.
• The use of spatial MaskGit for frame-wise refinement significantly improves visual quality compared to standard autoregressive heads, especially in high-resolution settings.
• Scaling the temporal transformer with increased model depth and width improves performance on long sequences, with 12-layer models achieving the best trade-off between quality and efficiency.