[Paper Review] Extrapolation and learning equations

Georg Martius, Christoph H. Lampert|arXiv (Cornell University)|Oct 10, 2016

Neural Networks and Applications18 citations

TL;DR

This paper introduces the Equation Learner (EQL), a differentiable feed-forward neural network that learns interpretable analytical expressions from data and generalizes to unseen domains through sparsity regularization and gradient-based training. EQL successfully extrapolates beyond training distributions when the true function lies within its hypothesis class, achieving high accuracy on synthetic and real-world physical systems.

ABSTRACT

In classical machine learning, regression is treated as a black box process of identifying a suitable function from a hypothesis set without attempting to gain insight into the mechanism connecting inputs and outputs. In the natural sciences, however, finding an interpretable function for a phenomenon is the prime goal as it allows to understand and generalize results. This paper proposes a novel type of function learning network, called equation learner (EQL), that can learn analytical expressions and is able to extrapolate to unseen domains. It is implemented as an end-to-end differentiable feed-forward network and allows for efficient gradient based training. Due to sparsity regularization concise interpretable expressions can be obtained. Often the true underlying source expression is identified.

Motivation & Objective

• To address the limitation of classical machine learning in extrapolating beyond training data distributions, especially in physical systems governed by analytical equations.
• To develop a function learning framework that produces interpretable, concise mathematical expressions rather than black-box approximations.
• To enable robust generalization to out-of-distribution inputs—critical for safety in robotics and control systems—by designing a model tailored for extrapolation.
• To introduce a novel model selection strategy based on multiobjective optimization of validation error and sparsity to favor generalizable, interpretable equations.
• To demonstrate that end-to-end differentiable architecture with structured nonlinear units can recover true underlying physical laws from data, even with limited or noisy observations.

Proposed method

• EQL is a multi-layer feed-forward network with linear transformations followed by specialized nonlinear units: unary functions (identity, sine, cosine, sigmoid) and binary multiplication units.
• The network's architecture is designed to represent common analytical forms found in physical equations, such as trigonometric and polynomial terms, via a fixed set of differentiable base functions.
• Sparsity is enforced through L1 regularization and L0 norm constraints on the weights, promoting compact, interpretable equations by eliminating irrelevant terms.
• The entire network is end-to-end differentiable, enabling backpropagation and efficient training using standard optimization techniques.
• Model selection uses a multiobjective criterion balancing empirical error on validation data and sparsity of the learned equation to prioritize generalization over interpolation.
• The method is evaluated on synthetic data with known ground-truth equations and real-world data from physical systems, including robotic dynamics and x-ray transition energies.

Experimental results

Research questions

• RQ1 Can a neural network architecture be designed to learn interpretable analytical expressions that generalize beyond the training distribution?
• RQ2 How does the inclusion of specific differentiable base functions (e.g., sine, cosine, sigmoid, multiplication) affect the model’s ability to recover true physical equations?
• RQ3 To what extent can sparsity regularization and L0 norm constraints improve the interpretability and extrapolation performance of learned equations?
• RQ4 How does the proposed model selection strategy based on validation error and sparsity compare to standard approaches in selecting models with strong extrapolation capabilities?
• RQ5 What are the limitations of the method when the true underlying function lies outside the hypothesis class of the network architecture?

Key findings

• EQL successfully recovers the true underlying equation in synthetic experiments when the target function is within the hypothesis class, demonstrating strong extrapolation performance.
• On the cart-pendulum system, which lies outside the hypothesis class due to rational functions (division), EQL and MLP show poor extrapolation, with EQL achieving a test error of 0.180 on the far extrapolation set.
• For the cart-pendulum system, the MLP outperforms EQL in near extrapolation (0.0184 vs. 0.0621), but both fail significantly in far extrapolation, indicating the model class is insufficient.
• In the case of the x-ray transition energy data, EQL learns a concise, interpretable equation that matches known physical relationships, demonstrating real-world applicability.
• The use of L1 regularization and L0 norm constraints leads to sparse, unbiased weight estimation, resulting in compact and interpretable equations that reflect the true functional form.
• The method fails to recover the correct equation when the true function is not representable within the network’s hypothesis class, highlighting the importance of expanding the base function set for broader applicability.