[Paper Review] Rule Covering for Interpretation and Boosting

Ş. İlker Birbil, Mert Edali+1 more|arXiv (Cornell University)|Jul 13, 2020

Neural Networks and Applications19 references4 citations

TL;DR

This paper proposes two mathematical programming-based algorithms for interpreting random forests and boosting decision trees using rule covering. The first algorithm extracts a minimal set of high-accuracy rules from a trained random forest, achieving performance close to the original model with far fewer rules. The second uses column generation with dual-based sample weighting to iteratively improve a base decision tree, outperforming standard boosting methods on multiple datasets.

ABSTRACT

We propose two algorithms for interpretation and boosting of tree-based ensemble methods. Both algorithms make use of mathematical programming models that are constructed with a set of rules extracted from an ensemble of decision trees. The objective is to obtain the minimum total impurity with the least number of rules that cover all the samples. The first algorithm uses the collection of decision trees obtained from a trained random forest model. Our numerical results show that the proposed rule covering approach selects only a few rules that could be used for interpreting the random forest model. Moreover, the resulting set of rules closely matches the accuracy level of the random forest model. Inspired by the column generation algorithm in linear programming, our second algorithm uses a rule generation scheme for boosting decision trees. We use the dual optimal solutions of the linear programming models as sample weights to obtain only those rules that would improve the accuracy. With a computational study, we observe that our second algorithm performs competitively with the other well-known boosting methods. Our implementations also demonstrate that both algorithms can be trivially coupled with the existing random forest and decision tree packages.

Motivation & Objective

• To develop a method for interpreting trained random forest models by extracting a minimal set of rules that preserve predictive accuracy.
• To create a boosting algorithm that selectively generates only the most effective rules to improve model performance.
• To integrate rule extraction and rule-based boosting into existing machine learning frameworks with minimal implementation overhead.
• To demonstrate that mathematical programming can simultaneously enhance model interpretability and predictive performance.

Proposed method

• The first algorithm formulates a mixed-integer linear program to select the fewest rules that cover all training samples while minimizing total impurity.
• Rules are extracted from the leaves of decision trees in a trained random forest and used as input to the mathematical programming model.
• The second algorithm uses column generation: at each iteration, a pricing subproblem identifies new rules that reduce the dual objective, using dual solutions as sample weights.
• Dual optimal solutions from the linear program are used to reweight misclassified samples, guiding the generation of high-impact rules in subsequent iterations.
• The algorithm iteratively improves model accuracy by solving a sequence of relaxed linear programs and adding only the most beneficial rules.
• Both algorithms are designed to be compatible with existing random forest and decision tree packages, enabling plug-in integration.

Experimental results

Research questions

• RQ1 Can a minimal set of rules extracted from a random forest model achieve predictive accuracy comparable to the full ensemble?
• RQ2 Can a rule-based boosting framework using mathematical programming outperform established boosting algorithms like AdaBoost and Gradient Boosting?
• RQ3 How effective is the use of dual solutions from linear programming as sample weights in guiding rule generation for boosting?
• RQ4 To what extent can rule covering reduce model complexity while preserving accuracy, enhancing interpretability?

Key findings

• The MIRCO algorithm achieved test accuracy comparable to random forests on 14 out of 15 datasets, with an average of less than 6% of test samples missed.
• MIRCO generated fewer rules than both random forests and decision trees on all but one dataset, significantly improving interpretability.
• RCBoost outperformed AdaBoost and Gradient Boosting on four datasets, demonstrating competitive performance with minimal rule generation.
• The number of RMP (pricing subproblem) calls in RCBoost was low—averaging under 50—indicating efficient convergence.
• The standard deviation of rule counts in MIRCO was lower than in decision trees on most datasets, indicating more stable rule set generation.
• Both algorithms were successfully implemented in Python and are compatible with existing machine learning libraries, enabling straightforward integration.