[Paper Review] Interpreting Blackbox Models via Model Extraction

Osbert Bastani, Carolyn Kim+1 more|arXiv (Cornell University)|May 23, 2017

Explainable Artificial Intelligence (XAI)Computer Science39 references131 citations

TL;DR

The paper presents a method to extract global, interpretable decision trees that closely approximate a blackbox model by actively sampling data; it evaluates fidelity and interpretability against baselines and demonstrates practitioner insights from the extracted trees.

ABSTRACT

Interpretability has become incredibly important as machine learning is increasingly used to inform consequential decisions. We propose to construct global explanations of complex, blackbox models in the form of a decision tree approximating the original model---as long as the decision tree is a good approximation, then it mirrors the computation performed by the blackbox model. We devise a novel algorithm for extracting decision tree explanations that actively samples new training points to avoid overfitting. We evaluate our algorithm on a random forest to predict diabetes risk and a learned controller for cart-pole. Compared to several baselines, our decision trees are both substantially more accurate and equally or more interpretable based on a user study. Finally, we describe several insights provided by our interpretations, including a causal issue validated by a physician.

Motivation & Objective

• Motivate interpretable global explanations as an alternative to local explanations or fully interpretable models.
• Propose a blackbox-to-decision-tree extraction algorithm that actively samples data to avoid overfitting.
• Show that the extracted decision trees closely approximate the blackbox model across tasks and are as or more interpretable in user studies.
• Demonstrate practical insights from interpretations, including clinician-validated causal observations and provider-dependent covariate shifts.

Proposed method

• Model extraction of axis-aligned decision trees to approximate a blackbox function f.
• Active sampling: iteratively sample x from an estimated input distribution P within the current node’s feasible region to label with f(x).
• Estimate the input distribution P via a mixture of axis-aligned Gaussians fitted with EM.
• Use an exact greedy tree construction (infinite data) as a target, then approximate it with n samples per split to form a finite extractor.
• Split selection uses a Gini-impurity based gain G(i,t) based on conditional label distributions under P, with leaf labels chosen by majority probability under C_N and its sub-conjunctions.
• Provide sampling and pruning rules to handle constraints C (x_i in [s_i, t_i]) and to ensure tractable sampling from p_P given C.
• Prove consistency: as n → ∞, the extracted tree Ť converges to the exact greedy tree T* (ε, δ-exact with sufficient samples).

Experimental results

Research questions

• RQ1 Can a global, interpretable model (a decision tree) faithfully approximate a complex blackbox model?
• RQ2 Does active sampling enable higher fidelity and reduced overfitting compared to baseline extraction methods?
• RQ3 Are extracted decision trees as interpretable as or more interpretable than competing explanations (rule lists, decision sets) in user studies?
• RQ4 What kinds of insights (including non-causal or provider-dependent effects) can be drawn from the extracted trees when applied to real datasets?
• RQ5 Do the extracted explanations reveal model behavior or biases not readily captured by alternative interpretability methods?

Key findings

• The proposed active-sampling tree extraction achieves higher fidelity to the blackbox model than CART and Born Again Trees across benchmarks.
• Decision trees extracted by the method are as interpretable or more interpretable than baselines in a user study with ML graduate students.
• Across diabetes-risk and cart-pole tasks, the extracted trees retain accurate reflections of the blackbox model’s decisions (high fidelity) while enabling inspection of subpopulation effects and potential non-causal patterns.
• The approach reveals covariate-shift and provider-specific differences in diabetes risk models, as well as translation-invariance and reflection biases in a cart-pole policy, offering actionable diagnostic insights.
• The method demonstrates stability and consistency gains with more samples, outperforming the Born Again Trees in terms of node-consistency across extractions.