[Paper Review] Using Shape Constraints for Improving Symbolic Regression Models

Christian Haider, Fabrício Olivetti de França+2 more|arXiv (Cornell University)|Jul 20, 2021

Evolutionary Algorithms and Applications35 references4 citations

TL;DR

This paper proposes a multi-objective symbolic regression approach that integrates shape constraints—such as monotonicity, convexity, and non-negativity—into genetic programming using interval arithmetic to verify constraints. By employing NSGA-II and MOEA/D with soft penalties for constraint violations, the method improves model trustworthiness and extrapolation, particularly under high noise, though no statistically significant performance gain over single-objective methods was found.

ABSTRACT

We describe and analyze algorithms for shape-constrained symbolic regression, which allows the inclusion of prior knowledge about the shape of the regression function. This is relevant in many areas of engineering -- in particular whenever a data-driven model obtained from measurements must have certain properties (e.g. positivity, monotonicity or convexity/concavity). We implement shape constraints using a soft-penalty approach which uses multi-objective algorithms to minimize constraint violations and training error. We use the non-dominated sorting genetic algorithm (NSGA-II) as well as the multi-objective evolutionary algorithm based on decomposition (MOEA/D). We use a set of models from physics textbooks to test the algorithms and compare against earlier results with single-objective algorithms. The results show that all algorithms are able to find models which conform to all shape constraints. Using shape constraints helps to improve extrapolation behavior of the models.

Motivation & Objective

• To improve symbolic regression by incorporating domain-specific shape constraints (e.g., monotonicity, convexity) to ensure models reflect physical laws.
• To investigate whether shape constraints enhance model generalization and extrapolation performance, especially in noisy data settings.
• To evaluate multi-objective evolutionary algorithms (NSGA-II, MOEA/D) against single-objective approaches in handling trade-offs between model accuracy and constraint compliance.
• To assess the impact of soft penalty mechanisms on constraint violation and overall model quality using interval arithmetic for rigorous verification.

Proposed method

• Shape constraints are encoded mathematically using first and second-order partial derivatives (e.g., ∂f/∂xi ≥ 0 for non-decreasing behavior) over input domains.
• Constraint feasibility is evaluated using interval arithmetic to compute rigorous bounds on derivatives, ensuring conservative (pessimistic) verification of shape properties.
• A soft-penalty approach transforms the constrained symbolic regression problem into a multi-objective optimization task minimizing both training error (NMSE) and constraint violation penalties.
• Multi-objective algorithms NSGA-II and MOEA/D are used to evolve a Pareto front of solutions balancing accuracy and constraint compliance.
• The final model is selected from the Pareto front based on the best NMSE, enabling trade-off analysis without rejecting infeasible solutions prematurely.
• Experiments are conducted on physics-based symbolic regression problems from textbooks, with noise levels systematically varied to assess robustness.

Experimental results

Research questions

• RQ1 Can multi-objective symbolic regression with shape constraints improve model reliability and extrapolation performance under noisy training data?
• RQ2 How do NSGA-II and MOEA/D compare to single-objective algorithms in balancing accuracy and constraint satisfaction in symbolic regression?
• RQ3 Does the inclusion of shape constraints lead to statistically significant improvements in out-of-sample (extrapolation) performance?
• RQ4 How does the choice of constraint verification method (pessimistic interval arithmetic vs. optimistic sampling) affect scalability and solution quality?

Key findings

• All tested algorithms—including NSGA-II, MOEA/D, and single-objective variants—successfully produced models that satisfied all specified shape constraints.
• NSGA-II achieved the best average ranking in critical difference plots for out-of-sample performance, especially at 100% noise, though no statistically significant difference was found compared to single-objective methods.
• Shape constraints did not yield a consistent, statistically significant improvement in overall performance across all problem instances, though one instance showed markedly better extrapolation with constraints.
• The computational overhead of constraint checking via interval arithmetic was minimal, with derivative computation being the main additional cost, which remained manageable for low-dimensional problems.
• Multi-objective approaches enabled exploration of trade-offs between accuracy and constraint violation, offering a set of diverse, valid solutions rather than a single optimal model.
• The method outperformed deterministic shape-constrained polynomial regression in symbolic regression tasks, though the latter required computationally expensive hyperparameter tuning via grid search.