[Paper Review] BIM: Towards Quantitative Evaluation of Interpretability Methods with Ground Truth.

Mengjiao Yang, Been Kim|arXiv (Cornell University)|Jul 23, 2019

Explainable Artificial Intelligence (XAI)12 references23 citations

TL;DR

This paper introduces BIM, a framework for quantitative evaluation of feature attribution methods in machine learning by leveraging a synthetic dataset with known relative feature importance. It proposes three metrics to compare attributions across models and inputs, revealing that certain methods produce more false positive explanations, and releases the dataset, models, and metrics for reproducibility.

ABSTRACT

Interpretability is an important area of research for safe deployment of machine learning systems. One particular type of interpretability method attributes model decisions to input features. Despite active development, quantitative evaluation of feature attribution methods remains difficult due to the lack of ground truth: we do not know which input features are in fact important to a model. In this work, we propose a framework for Benchmarking Attribution Methods (BAM) with a priori knowledge of relative feature importance. BAM includes 1) a carefully crafted dataset and models trained with known relative feature importance and 2) three complementary metrics to quantitatively evaluate attribution methods by comparing feature attributions between pairs of models and pairs of inputs. Our evaluation on several widely-used attribution methods suggests that certain methods are more likely to produce false positive explanations---features that are incorrectly attributed as more important to model prediction. We open source our dataset, models, and metrics.

Motivation & Objective

• To address the lack of ground truth in evaluating feature attribution methods in machine learning.
• To develop a framework that enables quantitative comparison of attribution methods using known relative feature importance.
• To identify systematic biases in existing attribution methods, particularly false positive attributions.
• To provide open-source tools, including a dataset, trained models, and evaluation metrics, for reproducible benchmarking.

Proposed method

• Design a synthetic dataset with controlled, known relative feature importance to serve as ground truth.
• Train multiple models on this dataset with predefined importance rankings to create reference points for evaluation.
• Define three complementary metrics to compare attributions across pairs of models and pairs of inputs.
• Use model pairs with known importance differences to evaluate how well attribution methods detect these differences.
• Apply the metrics to assess attribution fidelity by measuring consistency with known importance rankings.
• Open-source the dataset, models, and metrics to enable community-wide benchmarking and reproducibility.

Experimental results

Research questions

• RQ1 Which attribution methods most accurately reflect known relative feature importance in model predictions?
• RQ2 How do attribution methods perform when comparing models with known differences in feature importance?
• RQ3 To what extent do existing methods produce false positive attributions—assigning importance to features that are not actually influential?
• RQ4 How do different evaluation metrics complement each other in detecting flaws in attribution methods?

Key findings

• Certain attribution methods exhibit a higher tendency to produce false positive explanations, incorrectly identifying non-important features as influential.
• The proposed metrics successfully detect discrepancies between true and attributed feature importance, especially in model pairs with known importance differences.
• The framework reveals that popular methods such as Integrated Gradients and Grad-CAM show varying degrees of reliability under controlled conditions.
• The open-source dataset and models enable consistent, reproducible evaluation of attribution techniques across different research settings.
• Evaluation results show that attribution methods are not uniformly reliable, with performance varying significantly depending on model architecture and input distribution.
• The framework demonstrates that quantitative evaluation of interpretability is feasible when ground truth feature importance is known in advance.