[Paper Review] LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery

Pingchuan Ma, Tsun-Hsuan Wang+6 more|arXiv (Cornell University)|May 16, 2024

Scientific Computing and Data Management8 citations

TL;DR

The paper proposes Scientific Generative Agent (SGA), a bilevel framework where an LLM generates discrete hypotheses (expressions) and a differentiable simulator optimizes continuous parameters, enabling constitutive-law discovery and molecular design.

ABSTRACT

Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter challenges in effectively simulating observational feedback and grounding it with language to propel advancements in physical scientific discovery. Conversely, human scientists undertake scientific discovery by formulating hypotheses, conducting experiments, and revising theories through observational analysis. Inspired by this, we propose to enhance the knowledge-driven, abstract reasoning abilities of LLMs with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework: LLMs act as knowledgeable and versatile thinkers, proposing scientific hypotheses and reason about discrete components, such as physics equations or molecule structures; meanwhile, simulations function as experimental platforms, providing observational feedback and optimizing via differentiability for continuous parts, such as physical parameters. We conduct extensive experiments to demonstrate our framework's efficacy in constitutive law discovery and molecular design, unveiling novel solutions that differ from conventional human expectations yet remain coherent upon analysis.

Motivation & Objective

• Motivate a unified, knowledge-driven approach to physical scientific discovery beyond domain-specific methods.
• Combine LLMs with differentiable simulations in a bilevel optimization to search discrete hypotheses and optimize continuous parameters.
• Demonstrate the framework on constitutive law discovery and molecular design to uncover novel, coherent solutions.

Proposed method

• Outer-level search: LLM proposes discrete expressions (E) and continuous parameter space (Theta) based on past simulation results; inner-level optimization: differentiable simulation optimizes continuous parameters (theta) within Theta to evaluate E.
• Formulate bilevel optimization: minimize L(y(E, Theta, theta_hat; Phi)) subject to simulation validity G(E, Theta; Phi)≤0 and theta_hat = argmin_theta L(y(theta; Phi, E)).
• Exploit-and-explore strategy: adjust LLM generation temperature to balance exploitation of known good solutions and exploration of new hypotheses.
• Two interaction modes: equation searching (LLM proposes equations and Theta) and entity searching (LLM proposes structures with Theta as constants).
• Differentiable inner optimization: gradients from Phi wrt theta guide inner optimization and produce feedback o for the LLM.

Figure 1: The overall pipeline of Scientific Generative Agent (SGA). Taking the constitutive law searching problem as an example, the input is an initial guess (a purely elastic material), and the output is another constitutive law optimized towards the ground-truth (weakly compressible fluid). The

Experimental results

Research questions

• RQ1 Can a bilevel framework combining LLMs and differentiable simulations effectively discover constitutive laws from motion data?
• RQ2 Can the approach design molecules with target quantum properties by jointly optimizing discrete structures and continuous coordinates?
• RQ3 How does the exploitation-exploration balance affect discovery efficiency and solution quality in physical sciences?
• RQ4 Do LLMs generalize across domains (constitutive law discovery and molecular design) with minimal prompt modification?

Key findings

• The method outperforms several baselines across constitutive law discovery and molecule design tasks.
• Bilevel optimization is key: removing bilevel degrades performance, while retaining it with exploration improves results.
• Exploitation-exploration balance improves success and quality, with a 1:3 exploit:explore ratio yielding better results in challenging tasks.
• The approach discovers novel, well-performing constitutive laws and molecular designs that are coherent upon expert review.
• GPT-4 generally outperforms other backbones, though some open-source LLMs excel in specific molecule-design tasks.
• Ablation shows joint LLM-simulation interaction yields better proposals iteratively.

Figure 2: Loss trends comparison. Loss of the best solution averaged across seeds at different iterations of LLM-driven optimization, where the shading shows the min/max value.