[Paper Review] Evaluating the Correctness of Explainable AI Algorithms for Classification

Orcun Yalcin, Xiuyi Fan+1 more|arXiv (Cornell University)|May 20, 2021

Explainable Artificial Intelligence (XAI)Computer Science27 references17 citations

TL;DR

This paper proposes a method to quantitatively evaluate the correctness of explainable AI (XAI) algorithms in binary classification by generating synthetic datasets with known explanation ground truth using formal grammars. It demonstrates that SHAP outperforms LIME in explanation accuracy, and both classification and explanation performance degrade with increasing dataset complexity, measured via G-complexity.

ABSTRACT

Explainable AI has attracted much research attention in recent years with feature attribution algorithms, which compute "feature importance" in predictions, becoming increasingly popular. However, there is little analysis of the validity of these algorithms as there is no "ground truth" in the existing datasets to validate their correctness. In this work, we develop a method to quantitatively evaluate the correctness of XAI algorithms by creating datasets with known explanation ground truth. To this end, we focus on the binary classification problems. String datasets are constructed using formal language derived from a grammar. A string is positive if and only if a certain property is fulfilled. Symbols serving as explanation ground truth in a positive string are part of an explanation if and only if they contributes to fulfilling the property. Two popular feature attribution explainers, Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), are used in our experiments.We show that: (1) classification accuracy is positively correlated with explanation accuracy; (2) SHAP provides more accurate explanations than LIME; (3) explanation accuracy is negatively correlated with dataset complexity.

Motivation & Objective

• To address the lack of ground truth for evaluating XAI explanation correctness in real-world datasets.
• To develop a systematic, quantitative benchmark for evaluating feature attribution methods like LIME and SHAP.
• To define and measure 'correct' explanations in binary classification using formal language theory.
• To investigate the relationship between model performance and explanation accuracy.
• To introduce G-complexity as a controllable measure of dataset complexity affecting explainability.

Proposed method

• Construct binary classification datasets using formal grammars, where positive strings satisfy a specific property defined by the grammar.
• Define explanation ground truth as the set of symbols (features) that contribute to fulfilling the property in a positive string.
• Generate datasets with controlled complexity by varying grammar structure, modeling G-complexity after Kolmogorov complexity.
• Use LIME and SHAP as model-agnostic explainers to produce feature importance scores for each instance.
• Evaluate explanation correctness using a k-accuracy metric, measuring the proportion of top-k features matching the ground truth.
• Correlate classification AUC with explanation k-accuracy across varying string lengths and alphabet sizes.

Experimental results

Research questions

• RQ1 Can synthetic datasets with known explanation ground truth be systematically generated for evaluating XAI methods?
• RQ2 Is there a measurable correlation between classification accuracy and explanation accuracy?
• RQ3 How do SHAP and LIME compare in explanation correctness under controlled, known ground truth?
• RQ4 How does dataset complexity, as measured by G-complexity, affect both classification and explanation performance?
• RQ5 Does the proposed evaluation framework reveal systematic differences in explanation fidelity between LIME and SHAP?

Key findings

• Classification accuracy is positively correlated with explanation accuracy, indicating that better-performing models tend to produce more accurate explanations.
• SHAP provides significantly more accurate explanations than LIME, with a 49% higher average k-accuracy across experiments.
• Explanation accuracy decreases with increasing dataset complexity, as measured by G-complexity, showing a negative correlation.
• For string length 30, SHAP achieves a k-accuracy of -0.93, while LIME achieves only -0.53, indicating a substantial performance gap.
• With an alphabet size of 4 and string length 25, SHAP maintains higher mean k-accuracy (0.49) than LIME (0.27), confirming consistent superiority.
• The results show that SHAP better satisfies theoretical consistency properties than LIME, particularly in preserving identity and stability across similar inputs.