[Paper Review] PermuteAttack: Counterfactual Explanation of Machine Learning Credit Scorecards

Masoud Hashemi, Ali Fathi|arXiv (Cornell University)|Aug 24, 2020

Adversarial Robustness in Machine Learning23 references30 citations

TL;DR

Introduces PermuteAttack, a gradient-free, genetic-algorithm method to generate realistic counterfactuals for tabular credit-scoring models, enabling instance-level explanations and model critique.

ABSTRACT

This paper is a note on new directions and methodologies for validation and explanation of Machine Learning (ML) models employed for retail credit scoring in finance. Our proposed framework draws motivation from the field of Artificial Intelligence (AI) security and adversarial ML where the need for certifying the performance of the ML algorithms in the face of their overwhelming complexity poses a need for rethinking the traditional notions of model architecture selection, sensitivity analysis and stress testing. Our point of view is that the phenomenon of adversarial perturbations when detached from the AI security domain, has purely algorithmic roots and fall within the scope of model risk assessment. We propose a model criticism and explanation framework based on adversarially generated counterfactual examples for tabular data. A counterfactual example to a given instance in this context is defined as a synthetically generated data point sampled from the estimated data distribution which is treated differently by a model. The counterfactual examples can be used to provide a black-box instance-level explanation of the model behaviour as well as studying the regions in the input space where the model performance deteriorates. Adversarial example generating algorithms are extensively studied in the image and natural language processing (NLP) domains. However, most financial data come in tabular format and naive application of the existing techniques on this class of datasets generates unrealistic samples. In this paper, we propose a counterfactual example generation method capable of handling tabular data including discrete and categorical variables. Our proposed algorithm uses a gradient-free optimization based on genetic algorithms and therefore is applicable to any classification model.

Motivation & Objective

• Motivate model validation and explanation for ML-based retail credit scoring in Finance.
• Propose a black-box counterfactual generation framework for tabular data.
• Ensure counterfactuals are realistic and in-distribution by modeling higher-order interactions.
• Provide visual and statistical tools to analyze feature interactions and model sensitivity.

Proposed method

• Use a gradient-free genetic algorithm (PermuteAttack) to search for counterfactuals by permuting feature values picked from training data.
• Define a proximity-based fitness function that favors minimal perturbations and targeted class changes.
• Perturb features via permutation rather than small additive noise to preserve data validity for tabular data, including discrete/categorical variables.
• Apply one-hot encoding handling with ordinal conversion for permutation, followed by re-conversion for evaluation.
• Incorporate higher-order statistics via discretization and Gibbs sampling to sample from realistic conditional distributions.
• Report counterfactual instances and highlight changed features to explain model decisions.

Experimental results

Research questions

• RQ1 Can counterfactual explanations for tabular credit scoring be generated without model gradients?
• RQ2 Do permutation-based counterfactuals respect data distribution, including discrete and categorical features?
• RQ3 What is the typical feature-set size needed to flip predictions, and how do SHAP-identified important features align with perturbation frequencies?
• RQ4 How do counterfactual explanations relate to feature interactions in influencing model decisions?
• RQ5 Can the approach reveal potential fairness or discrimination signals by analyzing perturbations of sensitive attributes?

Key findings

• PermuteAttack generated counterfactuals for German Credit data with 100% success in flipping predictions under the tested hyperparameters.
• On average 2.35 features needed to be changed to flip a prediction.
• The most perturbed features align well with SHAP feature importance, though exact ordering varies across features.
• A co-change graph shows how features are perturbed together, indicating strong interactions among important features such as property, credit_history, and account_check_status.
• Limiting perturbations to sensitive attributes (age, personal_status_sex, foreign_worker) yields 26.25% success, indicating potential fairness considerations.
• Removing top SHAP features increases the average number of features required to change the outcome to 3.7, highlighting reliance on multiple attributes for flips.
• Gibbs sampling improves realism of counterfactuals; a discriminator trained to distinguish real vs counterfactual examples detects more counterfactuals when Gibbs sampling is used (49% success) than without (12%).