[Paper Review] SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretability

Wei Huang, Xingyu Zhao+2 more|arXiv (Cornell University)|Aug 19, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes SAFARI, a black-box evaluation framework for assessing the robustness of Deep Learning interpretability using two novel metrics: worst-case interpretation discrepancy and probabilistic interpretation robustness. It employs Genetic Algorithm with a bespoke fitness function and Subset Simulation to efficiently detect rare misinterpretations, outperforming state-of-the-art methods in accuracy, sensitivity, and efficiency across diverse XAI techniques and datasets.

ABSTRACT

Interpretability of Deep Learning (DL) is a barrier to trustworthy AI. Despite great efforts made by the Explainable AI (XAI) community, explanations lack robustness -- indistinguishable input perturbations may lead to different XAI results. Thus, it is vital to assess how robust DL interpretability is, given an XAI method. In this paper, we identify several challenges that the state-of-the-art is unable to cope with collectively: i) existing metrics are not comprehensive; ii) XAI techniques are highly heterogeneous; iii) misinterpretations are normally rare events. To tackle these challenges, we introduce two black-box evaluation methods, concerning the worst-case interpretation discrepancy and a probabilistic notion of how robust in general, respectively. Genetic Algorithm (GA) with bespoke fitness function is used to solve constrained optimisation for efficient worst-case evaluation. Subset Simulation (SS), dedicated to estimate rare event probabilities, is used for evaluating overall robustness. Experiments show that the accuracy, sensitivity, and efficiency of our methods outperform the state-of-the-arts. Finally, we demonstrate two applications of our methods: ranking robust XAI methods and selecting training schemes to improve both classification and interpretation robustness.

Motivation & Objective

• To address the lack of comprehensive, versatile, and efficient evaluation metrics for interpretability robustness in Deep Learning.
• To overcome the challenge of evaluating heterogeneous XAI techniques, especially perturbation-based methods, without requiring internal model access.
• To enable effective detection of rare misinterpretation events—common in local norm-balls—using black-box optimization and statistical sampling.
• To provide a holistic evaluation framework that complements worst-case and probabilistic robustness perspectives.
• To demonstrate practical applications: ranking XAI methods and identifying training schemes that improve both classification and interpretation robustness.

Proposed method

• Proposes two complementary robustness metrics: worst-case interpretation discrepancy (measured via PCC) and probabilistic robustness (estimated as the probability of misinterpretation within a norm ball).
• Uses a black-box Genetic Algorithm (GA) with a custom fitness function to solve constrained optimization and identify adversarial perturbations that maximize interpretation discrepancy.
• Employs Subset Simulation (SS), a statistical method for rare event probability estimation, to efficiently compute the likelihood of misinterpretation in local neighborhoods.
• Designs a bespoke fitness function in GA to guide search toward adversarial perturbations that cause maximal explanation change while preserving model prediction.
• Adapts SS to estimate low-probability misinterpretation events by iteratively sampling from conditional distributions, improving efficiency over standard Monte Carlo.
• Applies the framework across diverse datasets (MNIST, CIFAR-10, CelebA, ImageNet) and XAI methods (Gradient×Input, Integrated Gradients, GradCAM, FullGrad, Extremal Perturbations).

Experimental results

Research questions

• RQ1 How can we comprehensively evaluate the robustness of Deep Learning interpretability beyond binary adversarial detection?
• RQ2 What black-box methods can efficiently detect rare misinterpretation events in local norm-balls without requiring gradients or model internals?
• RQ3 Which XAI techniques are most robust to both perturbations that preserve and those that alter model predictions?
• RQ4 To what extent is interpretation robustness correlated with classification robustness across different neural network architectures?
• RQ5 Can we identify training schemes that simultaneously improve both classification and interpretation robustness?

Key findings

• Integrated Gradients consistently showed the highest robustness to misinterpretation across different architectures on CIFAR-10, maintaining low worst-case discrepancy and high probabilistic robustness.
• The probabilistic robustness metric (ln P) for Integrated Gradients was -35.93 on ResNet20 and -47.52 on VGG16, indicating strong resistance to misinterpretation.
• A strong correlation was empirically observed between classification robustness and interpretation robustness, suggesting shared underlying causes such as loss function curvature.
• FullGrad outperformed individual methods by combining layer-wise gradients, achieving a PCC of 0.799 and ln P of -75.716 on ImageNet, indicating high robustness.
• The GA-based worst-case evaluation achieved high sensitivity and efficiency, detecting adversarial perturbations with minimal computational overhead.
• Subset Simulation successfully estimated rare misinterpretation probabilities with high accuracy, outperforming standard Monte Carlo sampling in terms of computational cost.