[Paper Review] On the Safety of Interpretable Machine Learning: A Maximum Deviation Approach

Dennis Wei, Rahul Nair+4 more|arXiv (Cornell University)|Nov 2, 2022

Explainable Artificial Intelligence (XAI)6 citations

TL;DR

This paper introduces a maximum deviation approach to quantitatively assess the safety of interpretable machine learning models by measuring the largest output difference between a model under evaluation and a reference 'safe' model over a certification set. It demonstrates that interpretability enables exact or tight bounds on this deviation for models like decision trees, generalized linear models, and tree ensembles, with tighter regret bounds for piecewise Lipschitz functions, offering a principled way to quantify model safety and guide model selection.

ABSTRACT

Interpretable and explainable machine learning has seen a recent surge of interest. We focus on safety as a key motivation behind the surge and make the relationship between interpretability and safety more quantitative. Toward assessing safety, we introduce the concept of maximum deviation via an optimization problem to find the largest deviation of a supervised learning model from a reference model regarded as safe. We then show how interpretability facilitates this safety assessment. For models including decision trees, generalized linear and additive models, the maximum deviation can be computed exactly and efficiently. For tree ensembles, which are not regarded as interpretable, discrete optimization techniques can still provide informative bounds. For a broader class of piecewise Lipschitz functions, we leverage the multi-armed bandit literature to show that interpretability produces tighter (regret) bounds on the maximum deviation. We present case studies, including one on mortgage approval, to illustrate our methods and the insights about models that may be obtained from deviation maximization.

Motivation & Objective

• To provide a quantitative framework for assessing the safety of supervised machine learning models.
• To formalize safety as the maximum deviation of a model’s predictions from a reference model deemed safe.
• To investigate how interpretability enables more efficient and precise computation of this maximum deviation.
• To evaluate the effectiveness of the maximum deviation approach across different model classes, including interpretable and non-interpretable models.
• To demonstrate practical insights through case studies on mortgage approval data, revealing feature combinations that lead to extreme model outputs.

Proposed method

• Formulates safety assessment as an optimization problem: maximize the deviation D(f(x), f₀(x)) over a certification set C, where f is the model under evaluation and f₀ is the reference safe model.
• Uses a deviation measure D, such as L2 or absolute difference, to quantify output discrepancies between models.
• For interpretable models like decision trees, generalized linear models, and additive models, computes the maximum deviation exactly using structural properties.
• For tree ensembles, applies discrete optimization techniques to derive anytime upper and lower bounds on the maximum deviation.
• Generalizes to piecewise Lipschitz functions and leverages multi-armed bandit regret bounds to show that interpretability leads to significantly tighter bounds than for general black-box functions.
• Employs a certification set C, a subset of the input space covering plausible inputs, to define the domain over which deviation is maximized.

Experimental results

Research questions

• RQ1 How can model safety be formally and quantitatively defined in terms of deviation from a reference model?
• RQ2 To what extent does interpretability enable more efficient and accurate computation of maximum deviation in safety assessment?
• RQ3 How do the maximum deviation bounds scale with model complexity and certification set size?
• RQ4 Can informative bounds on maximum deviation be derived for non-interpretable models like tree ensembles?
• RQ5 What feature combinations lead to the largest deviations, and what insights do they provide for model behavior and domain validation?

Key findings

• For decision trees, generalized linear models, and additive models, the maximum deviation can be computed exactly and efficiently due to their structural interpretability.
• For tree ensembles, discrete optimization techniques yield informative upper and lower bounds on the maximum deviation, showing that the bound does not increase with the number of trees.
• In the Lending Club mortgage approval case study, the maximum deviation increases with the certification set radius r, with feature values like dti > 800% and annual_inc = $100 appearing at large r, indicating extreme, potentially unsafe inputs.
• The feature combinations that maximize deviation—such as high debt-to-income ratios, low income, long loan terms, and high-interest purposes—reveal model vulnerabilities and align with domain expectations.
• The maximum deviation for tree-based models is sensitive to model complexity, with higher ℓ₁ norm in logistic regression or more bins in Explainable Boosting Machines leading to larger deviations.
• The analysis reveals that even when post-hoc explanations suggest local faithfulness, the global behavior of models like tree ensembles may still exhibit large deviations from the reference model, highlighting the need for global safety assessment.