[Paper Review] AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity

Silviu‐Marian Udrescu, Andrew Y. Y. Tan+4 more|arXiv (Cornell University)|Jun 18, 2020

Model Reduction and Neural Networks36 references85 citations

TL;DR

An improved symbolic regression method that finds Pareto-optimal formulas by exploiting graph modularity, using gradient-based symmetry detection, Pareto-frontier pruning, hypothesis testing, and normalizing flows to handle distributions from samples.

ABSTRACT

We present an improved method for symbolic regression that seeks to fit data to formulas that are Pareto-optimal, in the sense of having the best accuracy for a given complexity. It improves on the previous state-of-the-art by typically being orders of magnitude more robust toward noise and bad data, and also by discovering many formulas that stumped previous methods. We develop a method for discovering generalized symmetries (arbitrary modularity in the computational graph of a formula) from gradient properties of a neural network fit. We use normalizing flows to generalize our symbolic regression method to probability distributions from which we only have samples, and employ statistical hypothesis testing to accelerate robust brute-force search.

Motivation & Objective

• Motivate the development of automated, interpretable symbolic regression for scientific data.
• Describe a modularity-aware divide-and-conquer regression framework that improves robustness to noise and outliers.
• Introduce a Pareto-frontier criterion to balance model simplicity and accuracy.
• Extend symbolic regression to learn distributions from samples using normalizing flows.

Proposed method

• Recursively decompose the mystery function using modularity in its computational graph inferred from neural-network gradients.
• Adopt a Pareto-frontier criterion based on description length to prune candidates by simplicity and accuracy (MEDL-based loss).
• Use statistical hypothesis testing to robustly reject non-frontier candidates instead of fixed thresholds.
• Employ normalizing flows to fit distributions from samples, enabling regression of probability densities.
• Train a fully-connected neural network to approximate the unknown function and test modularity with gradient-based tests (compositionality, generalized symmetry, generalized additivity).
• Implement a greedy, scalable search that prunes Pareto-dominated models after each merge step and applies parameter snapping and gradient-based refinement.

Experimental results

Research questions

• RQ1 Can gradient-based analysis reveal modular structure in the computation graph of the target function?
• RQ2 Does a Pareto-optimal, information-theoretic objective improve robustness to noise and outliers in symbolic regression?
• RQ3 Can normalizing flows extend symbolic regression to learn probability distributions from samples?
• RQ4 How does the method scale with input dimensionality and complexity of the target expression?
• RQ5 What are the success and failure modes of modularity-based symbolic regression on physics-inspired equations?

Key findings

• The method achieves robust symbolic regression by solving 73 of 100 baseline problems with Gaussian noise of standard deviation 10^(-1) (r = -1).
• It solves additional mysteries that prior work could not, including all 17 problems tackled by Schmidt & Lipson (2009) and additional cases, within at most two hours of runtime.
• Across new test equations with various graph modularities (T, S, P, G, M, C, A), the approach discovers correct forms and parameters, often matching underlying symmetries.
• Using normalizing flows, the method can regress probability distributions from samples, solving 80% of distributions listed in Table 5 with 10^2 to 10^5 samples.
• The approach improves robustness by relying on MEDL-based losses and Pareto-frontier pruning, reducing susceptibility to outliers and noise compared to prior methods.