[논문 리뷰] Bayesian Symbolic Regression

Ying Jin, Weilin Fu외 3명|arXiv (Cornell University)|2019. 10. 20.

Evolutionary Algorithms and Applications참고 문헌 27인용 수 47

한 줄 요약

Bayesian Symbolic Regression (BSR) 은 베이지안 프레임워크에서 기호 회귀를 적합시키며, 간결한 기호 트리의 가법 혼합과 MCMC를 사용해 사후 트리 구조를 샘플링하여 해석가능성과 견고성을 유전적 프로그래밍에 비해 향상시킵니다.

ABSTRACT

Interpretability is crucial for machine learning in many scenarios such as quantitative finance, banking, healthcare, etc. Symbolic regression (SR) is a classic interpretable machine learning method by bridging X and Y using mathematical expressions composed of some basic functions. However, the search space of all possible expressions grows exponentially with the length of the expression, making it infeasible for enumeration. Genetic programming (GP) has been traditionally and commonly used in SR to search for the optimal solution, but it suffers from several limitations, e.g. the difficulty in incorporating prior knowledge; overly-complicated output expression and reduced interpretability etc. To address these issues, we propose a new method to fit SR under a Bayesian framework. Firstly, Bayesian model can naturally incorporate prior knowledge (e.g., preference of basis functions, operators and raw features) to improve the efficiency of fitting SR. Secondly, to improve interpretability of expressions in SR, we aim to capture concise but informative signals. To this end, we assume the expected signal has an additive structure, i.e., a linear combination of several concise expressions, whose complexity is controlled by a well-designed prior distribution. In our setup, each expression is characterized by a symbolic tree, and the proposed SR model could be solved by sampling symbolic trees from the posterior distribution using an efficient Markov chain Monte Carlo (MCMC) algorithm. Finally, compared with GP, the proposed BSR(Bayesian Symbolic Regression) method saves computer memory with no need to keep an updated 'genome pool'. Numerical experiments show that, compared with GP, the solutions of BSR are closer to the ground truth and the expressions are more concise. Meanwhile we find the solution of BSR is robust to hyper-parameter specifications such as the number of trees.

연구 동기 및 목표

• Incorporate prior knowledge into symbolic regression to improve fitting efficiency and interpretability.
• Represent expressions as symbolic trees with controllable complexity via priors.
• Develop an MCMC-based posterior computation for tree structures and parameters.
• Demonstrate that BSR yields closer-to-ground-truth, more concise expressions than GP.
• Show robustness of BSR to hyper-parameter choices such as the number of additive components.

제안 방법

• Represent each mathematical expression as a symbolic tree with terminal nodes as features and non-terminal nodes as operators.
• Specify priors on the tree structure, terminal features, and linear transformation parameters to control complexity.
• Model the response as a linear combination of several concise expressions and estimate coefficients via OLS.
• Use Metropolis-Hastings and reversible jump MCMC to sample over tree structures and associated parameters.
• Apply RJMCMC to handle trans-dimensional changes when trees gain or lose lt() nodes.
• Compare BSR to GP on simulated benchmarks and a financial dataset to assess accuracy and interpretability.

실험 결과

연구 질문

• RQ1 Can Bayesian priors and an additive tree-structured representation yield more concise and interpretable symbolic expressions without sacrificing predictive accuracy?
• RQ2 How does BSR perform relative to Genetic Programming in terms of fit quality, generalization, and expression complexity across diverse tasks?
• RQ3 Is the BSR solution robust to the choice of the number of additive components (K) used in the model?

주요 결과

• BSR generally achieves closer-to-ground-truth expressions and shorter, more interpretable expressions than GP on benchmark tasks.
• BSR yields competitive RMSE on training and testing data, with notable improvements in several tasks.
• The additive tree structure results in significantly smaller expression sizes (fewer nodes) compared to GP across tasks.
• Increasing the number of additive components K can improve RMSE up to a point, after which gains diminish and redundant components are pruned by small coefficients.
• BSR demonstrated useful signals in real-world financial data, e.g., expressions associating open and low prices with next-day return signs.