[Paper Review] Global Model Interpretation via Recursive Partitioning

Chengliang Yang, Anand Rangarajan+1 more|arXiv (Cornell University)|Feb 11, 2018

Machine Learning and Data Classification30 references20 citations

TL;DR

This paper proposes Global Interpretation via Recursive Partitioning (GIRP), a method to generate a global, human-interpretable decision tree from local feature contribution matrices of black-box models. By recursively partitioning the input space to maximize differences in average feature contributions, GIRP produces a compact interpretation tree that reveals global decision rules, enabling model diagnosis and knowledge discovery in healthcare, vision, and NLP tasks with high fidelity to model behavior.

ABSTRACT

In this work, we propose a simple but effective method to interpret black-box machine learning models globally. That is, we use a compact binary tree, the interpretation tree, to explicitly represent the most important decision rules that are implicitly contained in the black-box machine learning models. This tree is learned from the contribution matrix which consists of the contributions of input variables to predicted scores for each single prediction. To generate the interpretation tree, a unified process recursively partitions the input variable space by maximizing the difference in the average contribution of the split variable between the divided spaces. We demonstrate the effectiveness of our method in diagnosing machine learning models on multiple tasks. Also, it is useful for new knowledge discovery as such insights are not easily identifiable when only looking at single predictions. In general, our work makes it easier and more efficient for human beings to understand machine learning models.

Motivation & Objective

• To address the gap between local and global model interpretation in machine learning, especially for complex, high-performing models.
• To develop a method that extracts globally applicable, human-understandable decision rules from black-box models without sacrificing predictive power.
• To enable efficient model diagnosis and new knowledge discovery by aggregating local explanations into a coherent global structure.
• To support population-level decision-making in high-stakes domains like healthcare and economics by revealing consistent patterns across millions of data samples.
• To provide a scalable, unified framework applicable to diverse ML models, including deep learning and ensemble methods.

Proposed method

• The method takes a contribution matrix as input, where each entry represents the contribution of a feature to the prediction for a single data sample.
• It recursively partitions the input space using a CART-like algorithm that selects splits maximizing the difference in average feature contributions between child nodes.
• At each split, the algorithm evaluates all features and all possible split points to find the one that maximizes the absolute difference in mean contributions between the two resulting subsets.
• The process continues until stopping criteria are met, such as minimum node size or no significant improvement in split quality.
• The resulting binary tree, called the interpretation tree, encodes global decision rules that summarize the model’s behavior across the entire dataset.
• The method captures feature interactions and heterogeneity in variable importance across subgroups by allowing splits to depend on combinations of features.

Experimental results

Research questions

• RQ1 Can a global, human-readable decision tree be constructed from local feature contribution matrices of diverse black-box models?
• RQ2 Does the recursive partitioning of feature contributions reveal meaningful, consistent decision rules that reflect the true behavior of complex models?
• RQ3 Can the interpretation tree detect model overfitting or reliance on spurious patterns, such as in text classification?
• RQ4 Can the method uncover novel, non-obvious relationships between input features and outcomes in real-world applications like ICU mortality prediction?
• RQ5 How well does the interpretation tree preserve the predictive behavior of the original model while remaining interpretable?

Key findings

• In computer vision, the interpretation tree correctly identified that a deep residual network focused on the intended object (e.g., a person in a scene) rather than spurious background features.
• In text classification, the tree revealed that a random forest model relied on irrelevant or misleading words (e.g., 'not' or 'but') to distinguish topics, indicating potential overfitting.
• In ICU mortality prediction, the method uncovered that 'convalescence and palliative care in 1 month' was a strong predictor of death (85.3% mortality), which aligns with clinical intuition.
• The method also identified 'other perinatal jaundice in 1 month' as a protective factor, suggesting a plausible clinical relationship not previously emphasized.
• The interpretation tree successfully captured heterogeneous effects, showing that feature importance varied meaningfully across different subgroups of patients.
• The approach enabled knowledge discovery by revealing previously unreported or underappreciated comorbidity patterns linked to ICU mortality.