[Paper Review] An Extensible Benchmark Suite for Learning to Simulate Physical Systems

Karl Otness, Arvi Gjoka+5 more|arXiv (Cornell University)|Aug 9, 2021

Model Reduction and Neural Networks50 references4 citations

TL;DR

This paper introduces an extensible benchmark suite for evaluating data-driven methods in physical simulation, featuring four representative physical systems, classical integrators, and diverse learning models (MLP, CNN, kernel, KNN). It reveals that even simple L²-based nearest neighbors outperform deep learning models in narrow data regimes, highlighting the high cost of ignoring physical priors in learning-based simulation.

ABSTRACT

Simulating physical systems is a core component of scientific computing, encompassing a wide range of physical domains and applications. Recently, there has been a surge in data-driven methods to complement traditional numerical simulations methods, motivated by the opportunity to reduce computational costs and/or learn new physical models leveraging access to large collections of data. However, the diversity of problem settings and applications has led to a plethora of approaches, each one evaluated on a different setup and with different evaluation metrics. We introduce a set of benchmark problems to take a step towards unified benchmarks and evaluation protocols. We propose four representative physical systems, as well as a collection of both widely used classical time integrators and representative data-driven methods (kernel-based, MLP, CNN, nearest neighbors). Our framework allows evaluating objectively and systematically the stability, accuracy, and computational efficiency of data-driven methods. Additionally, it is configurable to permit adjustments for accommodating other learning tasks and for establishing a foundation for future developments in machine learning for scientific computing.

Motivation & Objective

• To address the lack of standardized evaluation protocols for data-driven physical simulation methods.
• To provide a unified framework for objectively assessing stability, accuracy, and efficiency of learning-based simulators.
• To capture both narrow and wide data regimes to calibrate the difficulty of simulation tasks.
• To establish a modular, extensible infrastructure for future benchmarking in machine learning for scientific computing.
• To demonstrate that physics-agnostic models often fail to outperform simple baselines, especially in low-dimensional data regimes.

Proposed method

• The benchmark suite includes four representative physical systems: linear advection, elastic rods, incompressible Navier-Stokes, and a 2D fluid with solid obstacles.
• It provides reference implementations of classical time integrators (e.g., Runge-Kutta) as baseline numerical solutions.
• It integrates widely used data-driven models: MLPs, CNNs, kernel machines, and non-parametric nearest neighbors (KNN).
• The framework supports configurable initial condition sampling to model both narrow (low-dimensional manifold) and wide (high-dimensional) data regimes.
• A modular software stack enables dataset generation, model training, and evaluation using a consistent job description system with support for Slurm and Singularity.
• Evaluation is conducted via a three-phase pipeline: data generation, model training, and network evaluation with numerical integrators.

Experimental results

Research questions

• RQ1 How do data-driven simulation methods compare in accuracy, stability, and computational efficiency across diverse physical systems?
• RQ2 To what extent do deep learning models outperform simple baselines like nearest neighbors in physical simulation tasks?
• RQ3 How does the data regime (narrow vs. wide) influence the performance gap between physics-agnostic models and physics-informed numerical solvers?
• RQ4 Can a simple L²-based nearest neighbor regressor serve as a reliable calibration baseline for the difficulty of a physical simulation task?
• RQ5 What are the limitations of scaling up model size or dataset size in closing the performance gap between data-driven models and traditional numerical integration?

Key findings

• Even in the simplest physical models, data-driven methods fail to match the accuracy of traditional numerical integration, and this performance gap cannot be closed by scaling model size or dataset size.
• A simple L²-based nearest neighbor regressor outperforms most deep learning models in the narrow data regime, indicating that the learning task is often easier than assumed.
• The performance of deep learning models is highly sensitive to the data regime, with significant degradation in wide regimes where physical priors are essential.
• The study demonstrates that ignoring physical constraints leads to a high computational and accuracy cost, even for basic physical systems.
• The nearest neighbor baseline serves as a strong, interpretable calibration tool for assessing the true difficulty of a simulation task.
• The benchmark suite successfully enables systematic, reproducible evaluation across diverse physical systems and learning methods.