[Paper Review] The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations

Thibault Laugel, Marie‐Jeanne Lesot+3 more|arXiv (Cornell University)|Jul 22, 2019

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces a justification criterion for post-ho counterfactual explanations based on path-connectedness to training data, proposing a test to detect unjustified examples. It finds that state-of-the-art methods generate unjustified counterfactuals in up to 37% of cases on vulnerable instances, revealing a critical flaw in faithfulness to ground-truth data.

ABSTRACT

Post-hoc interpretability approaches have been proven to be powerful tools to generate explanations for the predictions made by a trained black-box model. However, they create the risk of having explanations that are a result of some artifacts learned by the model instead of actual knowledge from the data. This paper focuses on the case of counterfactual explanations and asks whether the generated instances can be justified, i.e. continuously connected to some ground-truth data. We evaluate the risk of generating unjustified counterfactual examples by investigating the local neighborhoods of instances whose predictions are to be explained and show that this risk is quite high for several datasets. Furthermore, we show that most state of the art approaches do not differentiate justified from unjustified counterfactual examples, leading to less useful explanations.

Motivation & Objective

• To define a justification criterion for counterfactual explanations based on continuous connectivity to training data.
• To develop a test to assess the risk of generating unjustified counterfactual examples in post-hoc interpretability.
• To evaluate the vulnerability of existing post-hoc counterfactual methods to generating unjustified explanations.
• To demonstrate that faithfulness to both the model and ground-truth data is often compromised in current approaches.
• To advocate for incorporating training data connectivity in future interpretability methods.

Proposed method

• Proposes a formal definition of a justified counterfactual explanation (JCF) as one that lies in a connected region of the same predicted class as a training instance.
• Introduces a verification procedure (VE) that checks whether a counterfactual example can be linked via an epsilon-chain to a training instance of the same predicted class.
• Uses hyperspherical layers centered on nearest neighbors to expand the search space for connected training instances.
• Applies the VE procedure to evaluate justification scores (J_E(x)) on counterfactuals generated by HCLS, Growing Spheres, and LORE.
• Measures the average justification score (J̄) across multiple instances and runs to assess method robustness.
• Employs a risk score (R_x) to identify instances with high local risk of generating unjustified counterfactuals (R_x ≥ 0.25).

Experimental results

Research questions

• RQ1 Can post-hoc counterfactual explanations be justified by connecting them to training data via continuous paths?
• RQ2 What is the risk of generating unjustified counterfactual examples in post-hoc methods, particularly in high-risk local neighborhoods?
• RQ3 How do state-of-the-art post-hoc counterfactual methods perform in generating justified examples?
• RQ4 To what extent does classifier complexity or data structure influence the justification of counterfactuals?
• RQ5 Can the connectedness of adversarial examples be analyzed under the same justification framework?

Key findings

• The proportion of justified counterfactuals (J̄) generated by state-of-the-art methods drops as low as 63% on instances with high risk (R_x ≥ 0.25).
• HCLS achieves slightly better justification performance than Growing Spheres, likely due to its focus on classification confidence over proximity.
• LORE achieves 100% justification on the Boston Random Forest and Recidivism datasets, indicating strong performance in specific settings.
• The justification rate varies significantly across datasets, with Iris showing 63% for both HCLS and GS, and BostonSVC reaching 95% for HCLS.
• The study confirms that post-hoc counterfactual methods are inherently vulnerable to generating unjustified examples due to lack of ground-truth data connectivity.
• The results suggest that current post-hoc methods cannot reliably ensure faithfulness to both the model and training data, necessitating future integration of training data in explanation generation.