[Paper Review] Generalized and Scalable Optimal Sparse Decision Trees

Jimmy Lin, Chudi Zhong+3 more|arXiv (Cornell University)|Jun 15, 2020

Explainable Artificial Intelligence (XAI)41 citations

TL;DR

GOSDT provides a general framework to optimize sparse decision trees for diverse objectives (e.g., AUC, F-score) and scalably handle continuous features without bucketization.

ABSTRACT

Decision tree optimization is notoriously difficult from a computational perspective but essential for the field of interpretable machine learning. Despite efforts over the past 40 years, only recently have optimization breakthroughs been made that have allowed practical algorithms to find optimal decision trees. These new techniques have the potential to trigger a paradigm shift where it is possible to construct sparse decision trees to efficiently optimize a variety of objective functions without relying on greedy splitting and pruning heuristics that often lead to suboptimal solutions. The contribution in this work is to provide a general framework for decision tree optimization that addresses the two significant open problems in the area: treatment of imbalanced data and fully optimizing over continuous variables. We present techniques that produce optimal decision trees over a variety of objectives including F-score, AUC, and partial area under the ROC convex hull. We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables and speeds up decision tree construction by several orders of magnitude relative to the state-of-the art.

Motivation & Objective

• Address imbalanced data and fully optimize over continuous variables in decision trees.
• Enable optimal sparse trees for multiple objectives (F-score, AUC, pAUC, weighted accuracy, balanced accuracy).
• Provide rigorous bounds and representations to prune the search space and certify optimality.

Proposed method

• Introduce a general dynamic programming with bounds (DPB) framework for sparse decision tree optimization.
• Develop new bounds (Hierarchical Objective Lower Bound, Incremental Progress Bounds, Similar Support Bounds, Equivalent Points Bound, Subset Bound) to prune the search space.
• Represent the DP searchspace via leaf sets and support sets to expose computational reuse for continuous features.
• Use a priority-queue driven asynchronous bound update mechanism to propagate improvements efficiently.
• Avoid bucketization; prove its optimality loss and directly optimize exact rank statistics (e.g., AUC, pAUC) on the training data.

Experimental results

Research questions

• RQ1 Can GOSDT optimally optimize sparse decision trees for non-accuracy objectives such as AUC and F-score?
• RQ2 How can continuous features be handled without bucketization while preserving optimality and speeding up computation?
• RQ3 What bounds and representations best reduce the search space and enable scalable optimization?
• RQ4 How does GOSDT perform on imbalanced datasets compared with existing optimal-tree methods?

Key findings

• GOSDT can optimize over a broad set of objectives including AUC, pAUC, F-score, balanced and weighted accuracy.
• Bucketization preprocessing degrades optimality; GOSDT preserves optimality by avoiding bucketization.
• The DPB-based GOSDT significantly improves scalability and runtime over prior methods, especially with continuous variables.
• A hierarchical lower bound and incremental/similar-support bounds dramatically prune the search space and enable faster optimality certificates.
• GOSDT achieves sparse, accurate trees and demonstrates time-to-optimality results on diverse datasets.