[Paper Review] Which priors matter? Benchmarking models for learning latent dynamics

Aleksandar Botev, Andrew Jaegle+3 more|arXiv (Cornell University)|Nov 9, 2021

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces a benchmark of 17 diverse physical dynamics datasets to rigorously evaluate models that incorporate physical priors—such as Hamiltonian and Lagrangian mechanics—for learning latent dynamics from high-dimensional observations. Despite the theoretical appeal of these priors, the study finds that standard Neural ODEs with continuous, time-reversible dynamics outperform physics-informed models, with continuity and time-reversibility emerging as the most critical inductive biases for long-horizon generalization.

ABSTRACT

Learning dynamics is at the heart of many important applications of machine learning (ML), such as robotics and autonomous driving. In these settings, ML algorithms typically need to reason about a physical system using high dimensional observations, such as images, without access to the underlying state. Recently, several methods have proposed to integrate priors from classical mechanics into ML models to address the challenge of physical reasoning from images. In this work, we take a sober look at the current capabilities of these models. To this end, we introduce a suite consisting of 17 datasets with visual observations based on physical systems exhibiting a wide range of dynamics. We conduct a thorough and detailed comparison of the major classes of physically inspired methods alongside several strong baselines. While models that incorporate physical priors can often learn latent spaces with desirable properties, our results demonstrate that these methods fail to significantly improve upon standard techniques. Nonetheless, we find that the use of continuous and time-reversible dynamics benefits models of all classes.

Motivation & Objective

• To evaluate whether physics-inspired inductive biases—particularly Hamiltonian and Lagrangian formalisms—improve learning of latent dynamics from high-dimensional visual data.
• To address the lack of standardized benchmarks and fair comparisons across models that claim to leverage physical priors.
• To identify which architectural inductive biases (e.g., continuous dynamics, time-reversibility) are most effective for long-horizon dynamics prediction.
• To release a reproducible, open-source benchmark suite to accelerate progress in learning dynamics from images.
• To assess the generalizability of physics-informed models across a wide range of dynamical systems, from toy physics to 3D environments.

Proposed method

• Designed a benchmark suite of 17 datasets spanning diverse physical systems, including mass-spring systems, molecular dynamics, 3D camera movements, and zero-sum games.
• Implemented and evaluated multiple models: Hamiltonian Graph Networks (HGN), Lagrangian Graph Networks (LGN), Neural ODEs, recurrent networks, and autoregressive models.
• Evaluated models using a VPT (Video Prediction Trajectory) score to measure long-horizon generalization, with trajectories extrapolated up to 1000 steps beyond training.
• Used consistent training and evaluation protocols across all models and datasets to ensure fair comparison.
• Trained models to predict both forward and backward in time to assess time-reversibility as an inductive bias.
• Employed a common framework for data generation and evaluation, with code and datasets open-sourced for reproducibility.

Experimental results

Research questions

• RQ1 Do models incorporating Hamiltonian or Lagrangian priors significantly outperform standard baselines in learning latent dynamics from images?
• RQ2 Which inductive biases—such as continuity, time-reversibility, or energy conservation—are most critical for long-horizon dynamics prediction?
• RQ3 How do physics-informed models perform across a diverse range of dynamical systems, from simple to complex?
• RQ4 Can the Hamiltonian manifold hypothesis be empirically validated using real-world-like visual observations and known equations of motion?
• RQ5 To what extent do physical priors reduce variance in long-horizon predictions compared to standard models?

Key findings

• Models incorporating physical priors such as Hamiltonian or Lagrangian formalisms do not significantly outperform standard Neural ODEs or baselines in most tasks.
• The most effective inductive bias across all model classes is continuity and time-reversibility of dynamics, which leads to more stable and consistent long-horizon predictions.
• Among physics-informed models, HGN (Hamiltonian Graph Networks) showed more consistent performance with lower variance in extrapolation than LGN (Lagrangian Graph Networks), which was more unstable and computationally costly.
• No model achieved reliable long-horizon extrapolation on the most complex dataset (MD-16), indicating current limitations in modeling complex molecular dynamics from images.
• The VPT score near 1.0 was achieved only on simpler datasets (e.g., mass-spring, two-body), indicating that these systems are effectively solved by at least one model, while more complex systems remain challenging.
• Despite theoretical advantages, direct incorporation of Hamiltonian/Lagrangian formalisms does not yet yield practical gains over standard Neural ODEs with continuous dynamics and residual learning.