[Paper Review] SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

Yadi Cao, Sicheng Lai+13 more|arXiv (Cornell University)|Mar 11, 2026

Scientific Computing and Data Management0 citations

TL;DR

SimulCost benchmarks cost-aware parameter tuning for physics simulations by evaluating LLMs against brute-force scan and Bayesian optimization across 12 simulators, measuring both success rate and computational cost.

ABSTRACT

Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,916 single-round (initial guess) and 1,900 multi-round (adjustment by trial-and-error) tasks across 12 simulators from fluid dynamics, solid mechanics, and plasma physics. Each simulator's cost is analytically defined and platform-independent. Frontier LLMs achieve 46--64% success rates in single-round mode, dropping to 35--54% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 71--80%, but LLMs are 1.5--2.5x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench.

Motivation & Objective

• Motivate the need for cost-aware evaluation in LLM-assisted physics simulations.
• Introduce SimulCost as the first benchmark to jointly measure success and tool-cost efficiency.
• Provide a diverse, extensible toolkit with 12 simulators and a reproducible cost-tracking framework.
• Compare state-of-the-art LLMs against brute-force scanning and Bayesian optimization.
• Offer ablations on knowledge transfer, in-context learning, and reasoning effort to guide deployment.

Proposed method

• Define cost as FLOPs-based tool cost for each simulator (with EPOCH using wall-clock time).
• Evaluate single-round (initial guess) and multi-round (trial-and-error) inference modes.
• Curate 2,916 single-round and 1,900 multi-round tasks across 12 solvers spanning fluid dynamics, solid mechanics, and plasma physics.
• Isolate tuning to individual parameters to enable meaningful scan baselines and cost comparisons.
• Offer an extensible toolbox (simulcost-tools) with standardized APIs and Hydra-based configuration for replication and extension.
• Include Bayesian Optimization as a baseline for multi-round tuning and perform ablations on ICL and reasoning effort.

Figure 1 : Overview of SimulCost . Our benchmark evaluates LLM agents on cost-sensitive parameter tuning across 12 physics simulators spanning fluid dynamics, solid mechanics, and plasma physics. Given a simulation task, tuning mode, and accuracy requirement, the LLM proposes tunable parameters in e

Experimental results

Research questions

• RQ1 How do LLMs perform in cost-aware parameter tuning for physics simulations across diverse simulators?
• RQ2 What is the trade-off between accuracy requirements and computational cost for single-round versus multi-round tuning?
• RQ3 Do knowledge transfer, in-context learning, or reasoning effort meaningfully improve cost-efficient tuning?
• RQ4 How does Bayesian optimization compare to LLM-based approaches in this cost-aware setting?
• RQ5 Can the toolkit generalize to new solvers and environments while preserving reproducible cost-tracking?

Key findings

• Frontier LLMs achieve 46–64% success in single-round mode, dropping to 35–54% under high accuracy requirements.
• Multi-round mode raises success to 71–80%, but LLMs are 1.5–2.5× slower than brute-force scanning.
• Common parameters are easier to tune than solver-specific ones, with little cross-parameter correlation suggesting limited transfer.
• In-context learning improves single-round success by 15–25% but degrades multi-round exploration.
• BO-GP matches aggregate success with higher inter-solver variance, while LLMs show cost-efficiency advantages at low accuracy demands.
• Reasoning effort overall shows no significant improvement.

((a)) Success Rate