[Paper Review] Exploring the Whole Rashomon Set of Sparse Decision Trees

Rui Xin, Chudi Zhong+4 more|PubMed|Sep 16, 2022

Machine Learning and AlgorithmsComputer Science6 references21 citations

TL;DR

The paper presents TreeFARMS, the first method to completely enumerate the Rashomon set of sparse decision trees, enabling querying, sampling, and analysis of all near-optimal models for nonlinear tree classes. It also demonstrates applications in variable importance, alternative metrics, and data-subset Rashomon sets.

ABSTRACT

In any given machine learning problem, there might be many models that explain the data almost equally well. However, most learning algorithms return only one of these models, leaving practitioners with no practical way to explore alternative models that might have desirable properties beyond what could be expressed by a loss function. The <i>Rashomon set</i> is the set of these all almost-optimal models. Rashomon sets can be large in size and complicated in structure, particularly for highly nonlinear function classes that allow complex interaction terms, such as decision trees. We provide the first technique for completely enumerating the Rashomon set for sparse decision trees; in fact, our work provides the first complete enumeration of any Rashomon set for a non-trivial problem with a highly nonlinear discrete function class. This allows the user an unprecedented level of control over model choice among all models that are approximately equally good. We represent the Rashomon set in a specialized data structure that supports efficient querying and sampling. We show three applications of the Rashomon set: 1) it can be used to study variable importance for the set of almost-optimal trees (as opposed to a single tree), 2) the Rashomon set for accuracy enables enumeration of the Rashomon sets for balanced accuracy and F1-score, and 3) the Rashomon set for a full dataset can be used to produce Rashomon sets constructed with only subsets of the data set. Thus, we are able to examine Rashomon sets across problems with a new lens, enabling users to choose models rather than be at the mercy of an algorithm that produces only a single model.

Motivation & Objective

• Motivate the study of all nearly optimal models (the Rashomon set) for sparse decision trees.
• Develop analytical bounds and a data-structure–driven method to enumerate the Rashomon set efficiently.
• Provide a scalable implementation (TreeFARMS) that stores, extracts, and samples Rashomon sets.
• Demonstrate practical uses: variable importance (MCR), Rashomon sets for alternative metrics, and robustness to data subsets.

Proposed method

• Define the epsilon-Rashomon set based on a reference objective threshold.
• Derive bounds (Basic Rashomon Lower Bound and Rashomon Equivalent Points Bound) to prune search space.
• Represent the Rashomon set with a hierarchical Model Set (MS) and Model Set Instances (MSI) to reduce duplication.
• Integrate with GOSDT to perform dynamic-programming with bounds and extract all near-optimal trees.
• Provide an extraction algorithm to build the MS from the GOSDT dependency graph (Algorithm 1 & 2).
• Offer a sampling mechanism to approximate the Rashomon set when full storage is infeasible (Section 4.3).

Experimental results

Research questions

• RQ1 What is the size and structure of the Rashomon set for sparse decision trees on real datasets?
• RQ2 How can variable importance vary across all well-performing models and how to measure it (MCR)?
• RQ3 Can Rashomon sets be constructed for metrics beyond accuracy, such as balanced accuracy and F1-score?
• RQ4 How does the Rashomon set change when subsets of data are removed or perturbed?

Key findings

• TreeFARMS substantially outperforms baselines, enumerating orders of magnitude more distinct Rashomon-set trees on multiple datasets.
• The method enables exact computation of Model Class Reliance (MCR) ranges, revealing variable importance across all well-performing trees.
• Rashomon sets for balanced accuracy and F1-score can be derived from the accuracy Rashomon set using provided theorems, enabling multi-metric analysis.
• Rashomon sets under data removal show high correlation of optimal trees, indicating robustness to small sample shifts.
• Sampling approaches can approximate MCR when full enumeration is infeasible, with 25% samples often sufficient for convergence.
• TreeFARMS guarantees complete enumeration of the Rashomon set for the considered sparse-tree class.