[Paper Review] How model accuracy and explanation fidelity influence user trust

Andrea Papenmeier, Gwenn Englebienne+1 more|arXiv (Cornell University)|Jul 26, 2019

Explainable Artificial Intelligence (XAI)21 references62 citations

TL;DR

The study shows that model accuracy more strongly drives user trust than explanation fidelity, and that explanations can harm trust if they are nonsensical or misaligned with a high-accuracy model; observed trust can diverge from self-reported trust.

ABSTRACT

Machine learning systems have become popular in fields such as marketing, financing, or data mining. While they are highly accurate, complex machine learning systems pose challenges for engineers and users. Their inherent complexity makes it impossible to easily judge their fairness and the correctness of statistically learned relations between variables and classes. Explainable AI aims to solve this challenge by modelling explanations alongside with the classifiers, potentially improving user trust and acceptance. However, users should not be fooled by persuasive, yet untruthful explanations. We therefore conduct a user study in which we investigate the effects of model accuracy and explanation fidelity, i.e. how truthfully the explanation represents the underlying model, on user trust. Our findings show that accuracy is more important for user trust than explainability. Adding an explanation for a classification result can potentially harm trust, e.g. when adding nonsensical explanations. We also found that users cannot be tricked by high-fidelity explanations into having trust for a bad classifier. Furthermore, we found a mismatch between observed (implicit) and self-reported (explicit) trust.

Motivation & Objective

• Examine how automatic decision system accuracy influences user trust.
• Investigate how presence and fidelity of explanations influence user trust.
• Evaluate the interaction between accuracy and explanation fidelity on trust.
• Validate a minimal-explanation approach with high and low fidelity.
• Provide empirical insights for explainable AI practitioners on trust dynamics.

Proposed method

• Develop nine classifier-explanation conditions combining three accuracy levels (high, medium, low) with three explanation fidelities (high, low, none).
• Use a social media moderation scenario classifying tweets as offensive or not.
• Train three classifiers: high accuracy (CNN-based), medium accuracy (logistic regression), and low accuracy (inverse-label CNN).
• Generate explanations: high-fidelity via L2X on CNN for high and inverse-label models; high-fidelity coefficients for medium; low-fidelity explanations via random word selections.
• Validate fidelity by reducing texts to selected features and checking if predictions match ground truth.
• Conduct a between-subject user study with 327 participants (286 valid) to measure self-reported trust and observed trust via behavior.
• Use a pre-study manual labeling task followed by system-assisted labeling on 15 tweets, then measure perceived understanding and trust using Körber (2018) questionnaire.
• Analyze trust data with Mann-Whitney U tests and Bonferroni correction for multiple comparisons.

Experimental results

Research questions

• RQ1 RQ1: What influence does the accuracy of an automatic decision system have on user trust?
• RQ2 RQ2: How do the presence and the level of fidelity of explanations influence user trust?
• RQ3 RQ3: How do accuracy and explanation fidelity interact to shape perceived understanding and trust dynamics?
• RQ4 RQ4: Do observed (behavioral) trust measures align with self-reported trust questionnaires?
• RQ5 RQ5: Can high-fidelity explanations mislead users about a poor classifier?

Key findings

• Accuracy had the strongest influence on user trust; higher accuracy generally yielded higher trust and predictability ratings.
• The presence of explanations did not positively affect self-reported trust in any condition, and high-fidelity explanations could maintain trust for medium accuracy but low-fidelity explanations lowered it.
• For high-accuracy systems, any explanation (high or low fidelity) tended to decrease trust compared to no explanations.
• For medium accuracy, high-fidelity explanations did not harm trust, while low-fidelity explanations did.
• Participants showed higher observed trust (behavior) for high-fidelity explanations with high accuracy than for no explanations, indicating a discrepancy between observed and self-reported trust.
• There is evidence that users are not fooled by a bad classifier even when paired with explanations.