[Paper Review] All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously

Aaron Fisher, Cynthia Rudin+1 more|arXiv (Cornell University)|Jan 4, 2018

Explainable Artificial Intelligence (XAI)Computer Science66 references1,156 citations

TL;DR

The paper introduces Model Class Reliance (MCR) and Model Reliance (MR) to quantify how much a covariate is used across all well-performing models in a predefined class, addressing the Rashomon effect by analyzing a whole set of near-optimal models rather than a single model.

ABSTRACT

Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})=\mathbf{x}^{T}β$ with a fixed coefficient vector $β$) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.

Motivation & Objective

• Motivate the Rashomon effect: multiple well-fitting models may rely on different covariates for prediction.
• Define MR to measure how much a single model relies on a covariate via a permutation-based loss-switching approach.
• Extend MR to MCR, the range of MR values across all well-performing models in a predefined class.
• Develop finite-sample bounds and practical computation methods for estimating MCR from data.
• Apply the framework to a public criminal justice data set to assess reliance on race, sex, or proxies in COMPAS scores.

Proposed method

• Define model class reliance (MCR) as the range of model reliance (MR) values across all models in a Rashomon set of near-optimal predictors.
• Define MR for a fixed model via the ratio of the loss when X1 is scrambled to the original loss.
• Bridge MR with U-statistics to establish unbiased estimators and asymptotic properties.
• Provide empirical MCR estimation via a general optimization procedure that yields a convex envelope containing the model class.
• Offer concrete implementations for (regularized) linear models and kernel-based models, with connections to additive models and conditional causal effects.
• Discuss finite-sample bounds and how bootstrap can be used to form practical confidence statements.

Experimental results

Research questions

• RQ1 How much can well-performing models rely on a covariate X1 while still achieving near-optimal predictive accuracy?
• RQ2 How can we summarize the variability in variable importance across all models within a Rashomon set?
• RQ3 What are the theoretical properties (unbiasedness, asymptotics) of MR and MCR estimators?
• RQ4 How can MR/MCR be computed in practice for common model classes like linear and kernel methods?
• RQ5 What do MR/MCR reveal about the role of sensitive attributes (e.g., race, sex) in proprietary predictive models like COMPAS?

Key findings

• MR quantifies an individual model’s reliance on X1 via a switch-based loss, and higher MR indicates greater dependence on X1.
• MCR extends MR to a range [MCR_-, MCR_+] across all well-performing models within a predefined class, capturing the Rashomon effect.
• MR estimators are unbiased and, under standard conditions, asymptotically normal as U-statistics; empirical MCR bounds provide meaningful finite-sample guarantees.
• A general optimization procedure yields a convex envelope that bounds empirical MCR for any epsilon (performance tolerance).
• Applications to Broward County recidivism data suggest that race, sex, and proxies for these variables are not necessarily the dominant predictors in the COMPAS score, illustrating the utility of MCR for scrutinizing proprietary models.