[Paper Review] LLM-SR: Scientific Equation Discovery via Programming with Large Language Models

Parshin Shojaee, Kazem Meidani+3 more|arXiv (Cornell University)|Apr 29, 2024

Scientific Computing and Data Management4 citations

TL;DR

LLM-SR introduces a novel framework that leverages Large Language Models (LLMs) to discover scientific equations by treating equations as executable programs, integrating scientific priors with data-driven optimization. It outperforms traditional symbolic regression baselines in discovering physically accurate equations across diverse scientific domains, demonstrating superior in-domain and out-of-domain generalization.

ABSTRACT

Mathematical equations have been unreasonably effective in describing complex natural phenomena across various scientific disciplines. However, discovering such insightful equations from data presents significant challenges due to the necessity of navigating extremely large combinatorial hypothesis spaces. Current methods of equation discovery, commonly known as symbolic regression techniques, largely focus on extracting equations from data alone, often neglecting the domain-specific prior knowledge that scientists typically depend on. They also employ limited representations such as expression trees, constraining the search space and expressiveness of equations. To bridge this gap, we introduce LLM-SR, a novel approach that leverages the extensive scientific knowledge and robust code generation capabilities of Large Language Models (LLMs) to discover scientific equations from data. Specifically, LLM-SR treats equations as programs with mathematical operators and combines LLMs' scientific priors with evolutionary search over equation programs. The LLM iteratively proposes new equation skeleton hypotheses, drawing from its domain knowledge, which are then optimized against data to estimate parameters. We evaluate LLM-SR on four benchmark problems across diverse scientific domains (e.g., physics, biology), which we carefully designed to simulate the discovery process and prevent LLM recitation. Our results demonstrate that LLM-SR discovers physically accurate equations that significantly outperform state-of-the-art symbolic regression baselines, particularly in out-of-domain test settings. We also show that LLM-SR's incorporation of scientific priors enables more efficient equation space exploration than the baselines. Code and data are available: https://github.com/deep-symbolic-mathematics/LLM-SR

Motivation & Objective

• Address the challenge of symbolic regression in high-dimensional, nonlinear hypothesis spaces by integrating scientific prior knowledge into equation discovery.
• Overcome limitations of traditional symbolic regression methods that neglect domain-specific knowledge and struggle with inefficient search in combinatorial spaces.
• Leverage the scientific reasoning and code generation capabilities of LLMs to generate informed equation skeletons and guide evolutionary search.
• Improve generalization and interpretability of discovered equations by embedding physical consistency and scientific plausibility through LLM priors.
• Develop a scalable, iterative framework that combines LLM-based hypothesis generation with data-driven evaluation and experience memory for refinement.

Proposed method

• Represent scientific equations as executable program skeletons using mathematical operators and symbolic variables, enabling LLMs to generate and manipulate them as code.
• Use structured prompts with problem descriptions and variable definitions to guide the LLM in generating scientifically plausible equation forms with placeholder coefficients.
• Employ off-the-shelf optimizers to fit the parameters of each equation skeleton to observed data, evaluating fitness via regression metrics.
• Maintain a dynamic memory buffer of high-scoring hypotheses to provide in-context examples that guide iterative refinement of LLM-generated equations.
• Apply evolutionary search strategies (mutation and crossover) guided by LLMs to explore the equation space efficiently, using fitness feedback to steer toward better solutions.
• Integrate scientific priors from LLMs—such as physical consistency, dimensional analysis, and known functional forms—into the hypothesis generation phase to improve search efficiency and accuracy.

Figure 1: The LLM-SR framework , consisting of three main steps: (a) Hypothesis Generation , where LLM generates equation program skeletons based on a structured prompt; (b) Data-driven Evaluation , which optimizes the parameters of each equation skeleton hypothesis and assesses its fit to the data;

Experimental results

Research questions

• RQ1 Can LLMs effectively generate scientifically plausible equation skeletons without relying solely on data-driven search?
• RQ2 How does incorporating scientific prior knowledge from LLMs improve the accuracy and generalization of discovered equations compared to traditional symbolic regression?
• RQ3 To what extent can LLM-generated equation programs outperform established baselines in fitting in-domain and out-of-domain data across diverse scientific domains?
• RQ4 How does the integration of experience memory and iterative refinement enhance the convergence and diversity of discovered equations?
• RQ5 What are the limitations of LLM-based equation discovery in terms of computational cost and potential biases in model-learned priors?

Key findings

• LLM-SR successfully discovers physically accurate equations across three distinct scientific domains, demonstrating strong generalization to out-of-domain data.
• The framework achieves significantly better fit to both in-domain and out-of-domain data compared to established symbolic regression baselines.
• The integration of LLM-generated scientific priors reduces the need for exhaustive search, accelerating convergence in high-dimensional equation spaces.
• The use of equation-as-program representation enables effective code generation and execution, enhancing expressiveness and flexibility in modeling complex relationships.
• Experience management via a dynamic memory of high-scoring hypotheses improves the quality of subsequent LLM-generated proposals through in-context learning.
• Despite computational costs, LLM-SR shows promise for accelerating scientific discovery by combining human-like reasoning with data-driven optimization.

Figure 2: Example of an input prompt for E. Coli bacterial growth with problem specifications, evaluation and optimization function, and the initial input equation example.