[Paper Review] CrevNet: Conditionally Reversible Video Prediction

Wei Yu, Yichao Lu+2 more|arXiv (Cornell University)|Oct 25, 2019

Advanced Image Processing Techniques9 references7 citations

TL;DR

CrevNet proposes a conditionally reversible video prediction model using a bijective two-way autoencoder and reversible predictive modules (RPMs) to achieve memory-efficient, information-preserving spatiotemporal modeling. By leveraging invertible architectures, CrevNet enables exact reconstruction from predictions, achieves state-of-the-art performance on high-resolution Traffic4cast with only 1 V100 GPU, and produces sharp, accurate video predictions without resolution-preserving blocks.

ABSTRACT

Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios. We propose CrevNet, a Conditionally Reversible Network that uses reversible architectures to build a bijective two-way autoencoder and its complementary recurrent predictor. Our model enjoys the theoretically guaranteed property of no information loss during the feature extraction, much lower memory consumption and computational efficiency.

Motivation & Objective

• To address the high memory and computational cost of resolution-preserving blocks in video prediction models.
• To explore the effectiveness of reversible architectures in video prediction, particularly for dense spatiotemporal generation tasks.
• To design a conditionally reversible framework that guarantees no information loss during feature extraction and prediction.
• To enable high-resolution video prediction on large-scale real-world datasets like Traffic4cast, which are infeasible for standard resolution-preserving models.

Proposed method

• CrevNet uses a two-way autoencoder based on additive coupling layers that perform bijective, volume-preserving transformations via forward and backward passes.
• The autoencoder splits input features channelwise, applies alternating residual-style updates with learnable convolutions, and ensures invertibility for exact reconstruction.
• A reversible predictive module (RPM) is introduced to extend reversibility into the temporal domain, using ConvRNN and a soft attention mechanism to model motion dynamics.
• The RPM employs a weighted sum of features and hidden states via learnable gates, enabling motion-aware feature propagation while preserving spatial alignment.
• The full model uses a single two-way autoencoder for both encoding and decoding, eliminating the need for skip connections or parameter sharing.
• Prediction is performed via the inverse pass of the autoencoder on features predicted by the RPM, enabling end-to-end conditional reversibility.

Experimental results

Research questions

• RQ1 Can invertible architectures be effectively applied to video prediction to reduce memory consumption while preserving information?
• RQ2 Does extending reversibility from spatial to temporal domains improve prediction quality and training stability?
• RQ3 Can a conditionally reversible model achieve state-of-the-art performance on high-resolution, real-world video prediction benchmarks like Traffic4cast?
• RQ4 How does the absence of resolution-preserving blocks affect performance on datasets with long-range motion and complex dynamics?

Key findings

• CrevNet achieves a per-pixel MSE of 9.251×10⁻³ on the Traffic4cast dataset, outperforming prior methods while training on a single V100 GPU.
• The model successfully handles high-resolution frames (495×436) and complex nonlinear dynamics, including long-range vehicle motion of up to 100 pixels per frame.
• The use of a single two-way autoencoder for both encoding and decoding reduces model parameters and eliminates the need for skip connections or feature sharing.
• The invertible architecture ensures no information loss during feature extraction, enabling sharp and accurate video predictions with fine-grained details.
• Fine-tuning on individual cities and timesteps further improves performance, reducing MSE from 9.392×10⁻³ to 9.251×10⁻³.