[Paper Review] Symbolic Regression for Constructing Analytic Models in Reinforcement Learning

Erik Derner, Jiřı́ Kubalı́k+2 more|arXiv (Cornell University)|Mar 27, 2019

Reinforcement Learning in Robotics37 references4 citations

TL;DR

This paper proposes using symbolic regression (SR) to automatically construct compact, interpretable analytic models from limited data for real-time reinforcement learning (RL) control. By applying SR to state-space and NARX-type input-output models, the method achieves superior performance over deep neural networks and local linear regression, enabling successful swing-up control on a real pendulum using only 100 data samples.

ABSTRACT

Reinforcement learning (RL) is a widely used approach for controlling systems with unknown or time-varying dynamics. Even though RL does not require a model of the system, it is known to be faster and safer when using models learned online. We propose to employ symbolic regression (SR) to construct parsimonious process models described by analytic equations for real-time RL control. We have tested our method with two different state-of-the-art SR algorithms which automatically search for equations that fit the measured data. In addition to the standard problem formulation in the state-space domain, we show how the method can also be applied to input-output models of the NARX (nonlinear autoregressive with exogenous input) type. We present the approach on three simulated examples with up to 14-dimensional state space: an inverted pendulum, a mobile robot, and a biped walking robot. A comparison with deep neural networks and local linear regression shows that SR in most cases outperforms these commonly used alternative methods. We demonstrate on a real pendulum system that the analytic model found enables RL to successfully perform the swing-up task, based on a model constructed from only 100 data samples.

Motivation & Objective

• To improve sample efficiency and safety in reinforcement learning by constructing interpretable, analytic process models from limited data.
• To address the challenge of modeling complex, nonlinear dynamics in systems with unknown or time-varying behavior.
• To enable real-time RL control by generating compact, closed-form equations that are computationally efficient and interpretable.
• To compare symbolic regression against deep neural networks and local linear regression in terms of accuracy, sample efficiency, and robustness.
• To validate the method on both simulated and real-world robotic systems, including high-dimensional state spaces.

Proposed method

• Employing two state-of-the-art symbolic regression algorithms to search for analytic equations that best fit measured system data.
• Formulating the system modeling problem in both state-space and NARX (nonlinear autoregressive with exogenous input) frameworks to capture dynamic behavior.
• Using data-driven discovery of equations to represent system dynamics, ensuring parsimony and interpretability.
• Training reinforcement learning agents using the symbolic models to perform control tasks such as swing-up and stabilization.
• Evaluating model performance by comparing prediction accuracy and RL control success across different model types.
• Applying the method to systems with up to 14-dimensional state spaces, including an inverted pendulum, mobile robot, and biped walker.

Experimental results

Research questions

• RQ1 Can symbolic regression generate accurate and compact analytic models from limited data for use in real-time reinforcement learning?
• RQ2 How does symbolic regression compare to deep neural networks and local linear regression in modeling nonlinear system dynamics?
• RQ3 Can symbolic models derived from only 100 data samples enable successful RL control in a real-world robotic system?
• RQ4 Does the NARX formulation improve model accuracy and control performance compared to standard state-space modeling?
• RQ5 To what extent do interpretable analytic models enhance sample efficiency and safety in RL-based control?

Key findings

• Symbolic regression consistently outperformed deep neural networks and local linear regression in modeling accuracy across all simulated examples.
• The method enabled successful swing-up control of a real pendulum using only 100 data samples to construct the analytic model.
• The resulting models were both parsimonious and interpretable, offering a significant advantage over black-box deep learning models.
• The approach was effective in high-dimensional systems, including a 14-dimensional biped walking robot, demonstrating scalability.
• The NARX formulation provided a viable alternative for input-output modeling, improving model fit in certain dynamic regimes.
• The use of symbolic models led to faster and more stable RL training compared to models learned via end-to-end neural networks.