[Paper Review] Model Class Reliance: Variable Importance Measures for any Machine Learning Model Class, from the "Rashomon" Perspective

Aaron Fisher, Cynthia Rudin+1 more|arXiv (Cornell University)|Jan 4, 2018

Explainable Artificial Intelligence (XAI)Computer Science119 citations

TL;DR

This paper introduces Model Class Reliance (MCR), a novel framework for variable importance that quantifies how much any model class, algorithm, or individual model relies on specific covariates by measuring changes in expected loss when a variable's value is switched. It provides probabilistic bounds and confidence regions for well-performing models, enabling comparison across diverse model types and revealing reliance patterns in real-world data, such as in recidivism prediction models using Broward County data.

ABSTRACT

Variable importance (VI) tools are typically used to examine the inner workings of prediction models. However, many existing VI measures are not comparable across model types, can obscure implicit assumptions about the data generating distribution, or can give seemingly incoherent results when multiple prediction models fit the data well. In this paper we propose a framework of VI measures for describing how much any model class (e.g. all linear models of dimension p), any model-fitting algorithm (e.g. Ridge regression with fixed regularization parameter), or any individual prediction model (e.g. a single linear model with fixed coefficient vector), relies on covariate(s) of interest. The building block of our approach, Model Reliance (MR), compares a prediction model's expected loss with that model's expected loss on a pair of observations in which the value of the covariate of interest has been switched. Expanding on MR, we propose Model Class Reliance (MCR) as the upper and lower bounds on the degree to which any well-performing prediction model within a class may rely on a variable of interest, or set of variables of interest. Thus, MCR describes reliance on a variable while accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. We give probabilistic bounds for MR and MCR, leveraging existing results for U-statistics. These bounds can be generalized to create finite-sample confidence regions for the best-performing models from any class. We also illustrate connections between MR, conditional causal effects, and linear regression coefficients. We outline a binary search procedure to compute estimates of MCR. We then apply MR & MCR in a public dataset of Broward County criminal records to study the reliance of recidivism prediction models on sex and race, with code available at https://github.com/aaronjfisher/mcr.

Motivation & Objective

• To address the lack of comparable variable importance (VI) measures across different model classes, such as linear models, tree-based models, or neural networks.
• To develop a framework that accounts for multiple well-fitting models within a class, avoiding reliance on implicit assumptions about the data-generating process.
• To provide probabilistic bounds and finite-sample confidence regions for the best-performing models in a class, enhancing reliability of VI estimates.
• To connect model reliance to conditional causal effects and linear regression coefficients, improving interpretability.
• To enable practical estimation of MCR via a binary search procedure and apply the method to real-world data, such as recidivism prediction in Broward County.

Proposed method

• Model Reliance (MR) is defined as the difference in expected loss between a model's predictions on original and swapped-pair observations, where the value of the covariate of interest is exchanged.
• Model Class Reliance (MCR) extends MR by computing upper and lower bounds on reliance across all well-performing models within a specified model class.
• The method leverages U-statistic theory to derive probabilistic bounds for MR and MCR, enabling finite-sample confidence regions for optimal models.
• A binary search procedure is proposed to efficiently estimate MCR by searching over the space of possible model parameters or coefficient vectors.
• The framework is applied to a public dataset of Broward County criminal records to analyze reliance on sex and race in recidivism prediction models.
• The approach is implemented in open-source code available on GitHub, facilitating reproducibility and broader adoption.

Experimental results

Research questions

• RQ1 How can variable importance be meaningfully compared across different model classes, such as linear models and random forests?
• RQ2 To what extent do well-performing models within a class rely on specific covariates, such as race or sex, when multiple models fit the data equally well?
• RQ3 Can MCR provide reliable, finite-sample confidence regions for the true reliance of the best-performing models in a class?
• RQ4 How does Model Reliance relate to conditional causal effects and linear regression coefficients in the presence of confounding or model misspecification?
• RQ5 What insights does MCR reveal about fairness and interpretability in real-world prediction systems, such as recidivism risk tools?

Key findings

• MCR provides upper and lower bounds on variable reliance across all well-performing models in a class, offering a robust alternative to single-model VI measures.
• The framework enables comparison of variable importance across model classes, such as linear models and tree-based models, without assuming a common parametric form.
• Probabilistic bounds derived from U-statistic theory allow for finite-sample confidence regions around MCR estimates, improving statistical reliability.
• The binary search procedure enables efficient estimation of MCR, making the method computationally feasible for high-dimensional settings.
• In the Broward County recidivism dataset, MCR revealed significant reliance on sex and race across well-fitting models, highlighting potential fairness concerns.
• The method connects to causal interpretation by relating Model Reliance to conditional causal effects, offering a path to interpretability beyond standard VI metrics.