[Paper Review] Gradient Information and Regularization for Gene Expression Programming to Develop Data-Driven Physics Closure Models

Fabian Waschkowski, Haochen Li+6 more|arXiv (Cornell University)|Nov 22, 2022

Evolutionary Algorithms and Applications4 citations

TL;DR

This paper enhances Gene Expression Programming (GEP) by integrating gradient-based optimization and regularization to develop accurate, interpretable, and implementable physics closure models. By introducing adaptive symbols that learn optimal numerical constants via nonlinear optimization and applying $L_2$ regularization and a custom complexity metric, the method significantly improves model accuracy and convergence speed across four use cases, including turbulence and combustion modeling.

ABSTRACT

Learning accurate numerical constants when developing algebraic models is a known challenge for evolutionary algorithms, such as Gene Expression Programming (GEP). This paper introduces the concept of adaptive symbols to the GEP framework by Weatheritt and Sandberg (2016) to develop advanced physics closure models. Adaptive symbols utilize gradient information to learn locally optimal numerical constants during model training, for which we investigate two types of nonlinear optimization algorithms. The second contribution of this work is implementing two regularization techniques to incentivize the development of implementable and interpretable closure models. We apply $L_2$ regularization to ensure small magnitude numerical constants and devise a novel complexity metric that supports the development of low complexity models via custom symbol complexities and multi-objective optimization. This extended framework is employed to four use cases, namely rediscovering Sutherland's viscosity law, developing laminar flame speed combustion models and training two types of fluid dynamics turbulence models. The model prediction accuracy and the convergence speed of training are improved significantly across all of the more and less complex use cases, respectively. The two regularization methods are essential for developing implementable closure models and we demonstrate that the developed turbulence models substantially improve simulations over state-of-the-art models.

Motivation & Objective

• Address the challenge of inaccurate numerical constants in evolutionary algorithms like GEP, which hinders the development of reliable physics closure models.
• Improve model interpretability and implementability by introducing regularization techniques to control model complexity and numerical constant magnitude.
• Develop a framework that combines symbolic regression flexibility with the precision of gradient-based optimization for data-driven model discovery.
• Demonstrate the framework's effectiveness across diverse physical systems, including fluid dynamics and combustion, to produce models that outperform state-of-the-art approaches.
• Enable robust and stable training by reducing sensitivity to random initialization through adaptive symbol design and hyperparameter tuning.

Proposed method

• Introduce adaptive symbols in GEP that use gradient information from nonlinear optimization (BFGS and Levenberg-Marquardt) to learn locally optimal numerical constants during training.
• Implement $L_2$ regularization to constrain the magnitude of learned numerical constants, preventing extreme values that compromise model stability.
• Propose a novel complexity metric based on custom symbol complexities to guide multi-objective optimization toward low-complexity, interpretable models.
• Apply the extended GEP framework to four use cases: Sutherland’s viscosity law, laminar flame speed modeling, and two turbulence closure models (SGS and NLEVM).
• Use a hybrid training strategy with multiple initializations, population sizes, and maximum runtimes tailored to each use case to ensure convergence and robustness.
• Employ multi-objective optimization to balance model accuracy, complexity, and numerical stability, with regularization parameters tuned via hyperparameter studies.

Experimental results

Research questions

• RQ1 Can gradient-informed optimization of numerical constants in GEP improve model accuracy and convergence speed compared to traditional evolutionary methods?
• RQ2 To what extent does $L_2$ regularization reduce the magnitude of learned constants and enhance model stability and implementability?
• RQ3 How effective is the proposed complexity metric in guiding the discovery of low-complexity, interpretable closure models?
• RQ4 Can the enhanced GEP framework rediscover known physical laws (e.g., Sutherland’s law) and outperform existing models in complex turbulence modeling tasks?
• RQ5 How do different optimization algorithms (BFGS vs. Levenberg-Marquardt) and adaptive symbol counts affect training reliability and performance?

Key findings

• The integration of gradient-based optimization via adaptive symbols reduced training error and improved convergence speed across all use cases, with the best performance achieved using five adaptive symbols per scalar function.
• For the NLEVM turbulence model, $L_2$ regularization with $λ = 10^{-7}$ reduced the maximum adaptive symbol magnitude from $8.55 \times 10^{17}$ to $1.12 \times 10^{1}$, significantly improving numerical stability.
• The model with $λ = 10^{-7}$ achieved a test MSE of 0.5770, outperforming unregularized training ($\text{MSE} = 0.5558$) in terms of stability and generalization, despite a slight increase in complexity.
• The use of the custom complexity metric and multi-objective optimization led to models with lower $J_c$ values, indicating improved interpretability and reduced computational burden.
• The developed NLEVM turbulence model substantially improved simulation accuracy over state-of-the-art models, demonstrating the framework’s practical utility in complex fluid dynamics.
• Hyperparameter studies confirmed that $n_p = 5$ adaptive symbols per scalar function provided the optimal balance between performance and robustness, minimizing sensitivity to random initialization.