[Paper Review] ERFit: Entropic Regression Fit Matlab Package, for Data-Driven System Identification of Underlying Dynamic Equations

Abd AlRahman R. AlMomani, Erik M. Bollt|arXiv (Cornell University)|Oct 5, 2020

Evolutionary Algorithms and Applications4 citations

TL;DR

ERFit is a MATLAB package for data-driven system identification using entropic regression, which employs conditional mutual information to iteratively select sparse, informative basis functions for modeling nonlinear dynamical systems. It combines forward selection and backward elimination to discover accurate, low-dimensional models from time-series data with minimal user supervision, achieving high accuracy on benchmarks like Lorenz and Rossler systems.

ABSTRACT

Data-driven sparse system identification becomes the general framework for a wide range of problems in science and engineering. It is a problem of growing importance in applied machine learning and artificial intelligence algorithms. In this work, we developed the Entropic Regression Software Package (ERFit), a MATLAB package for sparse system identification using the entropic regression method. The code requires minimal supervision, with a wide range of options that make it adapt easily to different problems in science and engineering. The ERFit is available at https://github.com/almomaa/ERFit-Package

Motivation & Objective

• To develop a user-friendly, minimal-supervision MATLAB package for sparse system identification in nonlinear dynamical systems.
• To address the challenge of identifying underlying dynamic equations from high-dimensional time-series data without prior knowledge of the system structure.
• To improve model accuracy and avoid overfitting by leveraging information-theoretic criteria over traditional least-squares or sparsity-promoting methods.
• To enable researchers to easily customize derivative estimation, mutual information computation, and parameter magnitude estimation for diverse scientific applications.

Proposed method

• ERFit uses entropic regression (ER), a greedy optimization method based on conditional mutual information to select relevant basis functions from a candidate library.
• The forward selection stage iteratively adds basis functions that maximize the conditional mutual information between the vector field and the candidate function, given the current model.
• The backward elimination stage removes redundant or non-informative functions by minimizing their conditional mutual information contribution to the model.
• The algorithm employs a termination condition (HLT1) based on a tolerance threshold for information gain, halting when no further significant improvement is possible.
• The method supports user-defined estimators for derivatives, mutual information, and parameter magnitudes, enhancing flexibility and adaptability.
• The package is implemented in MATLAB with full documentation and live scripts, ensuring accessibility and extensibility for researchers across disciplines.

Experimental results

Research questions

• RQ1 Can entropic regression with conditional mutual information outperform traditional sparse regression methods in identifying underlying dynamics from noisy time-series data?
• RQ2 How effective is the two-stage forward selection and backward elimination process in reducing model complexity while preserving predictive accuracy?
• RQ3 To what extent can ERFit discover correct dynamical equations from high-dimensional systems, such as networks of coupled oscillators or PDEs like Kuramoto-Sivashinsky?
• RQ4 How robust is the method to noise and high-dimensional basis function libraries in real-world scientific applications?
• RQ5 Can the modular design of ERFit support custom estimation strategies for derivative and information-theoretic quantities in diverse modeling contexts?

Key findings

• ERFit successfully identified the true underlying dynamics of the Lorenz and Rossler systems from time-series data, achieving high accuracy with minimal user input.
• The method demonstrated robust performance on high-dimensional systems, including a 300-dimensional network of coupled Lorenz systems, where the regression problem expanded to over 45,000 dimensions.
• The backward elimination stage effectively removed redundant basis functions, improving model sparsity and reducing overfitting without sacrificing predictive power.
• The package achieved accurate system identification on the 16-dimensional Kuramoto-Sivashinsky equation, with the regression problem expanding to approximately 150 dimensions due to basis function expansion.
• The algorithm's termination condition (HLT1) based on information gain tolerance effectively prevented unnecessary iterations, ensuring computational efficiency.
• ERFit is compatible with MATLAB R2016a and later, and supports user-defined components for derivative estimation, mutual information, and parameter magnitude, enabling broad scientific reuse.