[Paper Review] PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction

Hao Wu, Fan Xu+4 more|arXiv (Cornell University)|May 19, 2023

Advanced Image Processing Techniques4 citations

TL;DR

PastNet introduces physical inductive biases into spatio-temporal video prediction by leveraging spectral convolution in the Fourier domain to encode underlying physical laws, while using a memory bank with intrinsic dimensionality estimation to reduce computational cost. It achieves state-of-the-art performance in video prediction quality and training efficiency, particularly on high-resolution videos, with a training time of just 4.16 hours for 100 epochs on MovingMNIST and superior MSE, SSIM, and PSNR scores compared to prior methods.

ABSTRACT

In this paper, we investigate the challenge of spatio-temporal video prediction task, which involves generating future video frames based on historical spatio-temporal observation streams. Existing approaches typically utilize external information such as semantic maps to improve video prediction accuracy, which often neglect the inherent physical knowledge embedded within videos. Worse still, their high computational costs could impede their applications for high-resolution videos. To address these constraints, we introduce a novel framework called \underline{P}hysics-\underline{a}ssisted \underline{S}patio-\underline{t}emporal \underline{Net}work (PastNet) for high-quality video prediction. The core of PastNet lies in incorporating a spectral convolution operator in the Fourier domain, which efficiently introduces inductive biases from the underlying physical laws. Additionally, we employ a memory bank with the estimated intrinsic dimensionality to discretize local features during the processing of complex spatio-temporal signals, thereby reducing computational costs and facilitating efficient high-resolution video prediction. Extensive experiments on various widely-used spatio-temporal video benchmarks demonstrate the effectiveness and efficiency of the proposed PastNet compared with a range of state-of-the-art methods, particularly in high-resolution scenarios.

Motivation & Objective

• To address the limitation of existing video prediction models that neglect underlying physical laws governing spatio-temporal dynamics.
• To reduce computational complexity for high-resolution video prediction without sacrificing accuracy.
• To integrate physical inductive biases directly from video data, avoiding reliance on external inputs like semantic maps or optical flow.
• To enable efficient, scalable, and high-quality video prediction using a learnable spectral convolution mechanism in the Fourier domain.
• To demonstrate the model's capability in solving partial differential equations (PDEs) such as Navier-Stokes and Shallow-Water equations.

Proposed method

• Introduces a spectral convolution operator in the Fourier domain to embed physical inductive biases derived from the data itself.
• Applies an inverse Fourier transform after spectral convolution to reconstruct output frames in the spatial domain.
• Employs a discrete spatio-temporal module that estimates intrinsic dimensionality and uses memory banks to discretize local features, reducing computational load.
• Uses a learnable feature projection gate (FPG) module to enhance feature representation and adaptability across different datasets.
• Integrates a memory bank mechanism that replaces local features with compact, learned representations to improve efficiency.
• Trains end-to-end using standard video prediction loss functions, with no need for external supervision or auxiliary signals.

Figure 1. Performance comparison of different video prediction methods on MovingMNIST. Our PastNet outperforms previous models in terms of training time and image quality. Specifically, the PastNet achieves the lowest MSE and highest MS-SSIM scores when trained for 100 epochs, and its training time

Experimental results

Research questions

• RQ1 Can spectral convolution in the Fourier domain effectively encode physical inductive biases from raw video data for improved prediction?
• RQ2 Does the integration of intrinsic dimensionality estimation and memory banks significantly reduce computational cost while maintaining high prediction quality?
• RQ3 Can PastNet generalize across diverse datasets, including natural phenomena and synthetic video benchmarks, with consistent performance?
• RQ4 To what extent can PastNet solve complex PDEs such as Navier-Stokes and Shallow-Water equations in a video prediction setting?
• RQ5 How does PastNet compare in training speed and convergence rate to state-of-the-art models on high-resolution video prediction tasks?

Key findings

• PastNet achieves the lowest MSE and highest MS-SSIM on MovingMNIST after 100 epochs, with a training time of only 4.16 hours—significantly faster than competing models.
• On the Natural Phenomenon dataset, PastNet with the FPG module achieves the best performance, and using UNet as an alternative module maintains strong results, demonstrating flexibility.
• PastNet reaches an SSIM of approximately 0.92 within 60 epochs, while other models remain below 0.85, indicating a rapid convergence rate.
• After 100 epochs, PastNet attains a PSNR of around 38, substantially outperforming other models in image quality.
• PastNet demonstrates strong potential for solving PDEs, achieving lower MSE and MAE on the Shallow-Water Equation (SWE) than on the Navier-Stokes Equation (NSE), with similar training time per epoch.
• The ablation study confirms that both the FPG and DST modules are critical: removing FPG degrades performance, while removing DST slows training, highlighting their roles in accuracy and efficiency.

Figure 2. An overview of the proposed PastNet, which consists of a Fourier-based Physics-guided (FPG) and a Discrete Spatio-temporal (DST) module. The PFI module first divides video frames into non-overlapping patches and introduce Fourier-based spectral filter with the introduction of physical bias