[Paper Review] Frequency Domain Transformer Networks for Video Prediction

Hafez Farazi, Sven Behnke|arXiv (Cornell University)|Mar 1, 2019

Advanced Vision and Imaging27 references4 citations

TL;DR

This paper proposes Frequency Domain Transformer Networks (FDTN), an end-to-end learnable architecture that models video prediction by estimating temporal transformations via phase differences in the frequency domain, enabling more accurate and stable long-term forecasting. FDTN outperforms baselines like Conv-PGP and VLN on Moving MNIST and Bouncing Ball datasets with lower prediction loss and fewer parameters.

ABSTRACT

The task of video prediction is forecasting the next frames given some previous frames. Despite much recent progress, this task is still challenging mainly due to high nonlinearity in the spatial domain. To address this issue, we propose a novel architecture, Frequency Domain Transformer Network (FDTN), which is an end-to-end learnable model that estimates and uses the transformations of the signal in the frequency domain. Experimental evaluations show that this approach can outperform some widely used video prediction methods like Video Ladder Network (VLN) and Predictive Gated Pyramids (PGP).

Motivation & Objective

• To address the challenge of high nonlinearity in spatial-domain video prediction by leveraging frequency domain representations.
• To improve long-term video prediction accuracy by modeling motion transformations via phase differences in the Fourier domain.
• To design an end-to-end trainable architecture that explicitly estimates and applies temporal transformations in the frequency domain.
• To outperform existing methods like VLN and Conv-PGP on synthetic video prediction benchmarks.

Proposed method

• The model computes the Fast Fourier Transform (FFT) of input frames to represent them in the frequency domain.
• It estimates the transformation between consecutive frames as the element-wise phase difference of their complex frequency representations.
• A learnable 'Transform Model' processes the phase difference representation to predict changes in motion, using fully connected or convolutional layers with ReLU/sigmoid activations.
• The predicted transformation is applied via phase rotation in the frequency domain to generate the next frame's frequency representation.
• A 'Refine Model' with three convolutional layers with ReLU activations reconstructs high-frequency details in the spatial domain.
• The architecture uses a softmax-weighted fusion of multiple transformed representations (including flipped versions) to handle boundary effects and improve robustness.

Experimental results

Research questions

• RQ1 Can modeling motion transformations in the frequency domain improve long-term video prediction performance compared to spatial-domain methods?
• RQ2 How does phase difference estimation in the Fourier domain compare to traditional bilinear or gated autoencoder-based transformation modeling?
• RQ3 To what extent can a learnable frequency-domain transformation model generalize across different synthetic video dynamics like moving digits and bouncing balls?
• RQ4 Does the integration of a dedicated 'Refine Model' reduce reconstruction blur and improve perceptual quality in predicted frames?
• RQ5 How do architectural choices—fully connected vs. convolutional 'Transform Model'—affect performance and parameter efficiency?

Key findings

• FDTN(Conv) achieved a mean squared prediction loss of 0.00316 on the Moving MNIST dataset, outperforming Conv-PGP (0.06963) and VLN-ResNet (0.00544).
• FDTN(FC) achieved a loss of 0.00285 on Moving MNIST and 0.00086 on Bouncing Ball, significantly outperforming Conv-PGP and VLN-ResNet.
• The FDTN(FC) variant achieved the best performance despite having 160K parameters, compared to VLN-ResNet’s 1.3M parameters.
• Removing the 'Refine Model' led to blurry predictions, demonstrating its critical role in preserving high-frequency details.
• Eliminating the 'Transform Model' caused failure in modeling dynamic changes in motion, confirming its necessity for accurate transformation prediction.
• The model generalized well to longer sequences, maintaining performance beyond the 10-frame training horizon, unlike baselines.