[Paper Review] Explanations Can Reduce Overreliance on AI Systems During Decision-Making

Helena Vasconcelos, Matthew Jörke+4 more|arXiv (Cornell University)|Dec 13, 2022

Decision-Making and Behavioral Economics20 citations

TL;DR

The paper shows that AI explanations can reduce overreliance when the task and explanation costs are manipulated in a maze-solving collaboration with a simulated AI, using a cost-benefit framework across five studies (N=731).

ABSTRACT

Prior work has identified a resilient phenomenon that threatens the performance of human-AI decision-making teams: overreliance, when people agree with an AI, even when it is incorrect. Surprisingly, overreliance does not reduce when the AI produces explanations for its predictions, compared to only providing predictions. Some have argued that overreliance results from cognitive biases or uncalibrated trust, attributing overreliance to an inevitability of human cognition. By contrast, our paper argues that people strategically choose whether or not to engage with an AI explanation, demonstrating empirically that there are scenarios where AI explanations reduce overreliance. To achieve this, we formalize this strategic choice in a cost-benefit framework, where the costs and benefits of engaging with the task are weighed against the costs and benefits of relying on the AI. We manipulate the costs and benefits in a maze task, where participants collaborate with a simulated AI to find the exit of a maze. Through 5 studies (N = 731), we find that costs such as task difficulty (Study 1), explanation difficulty (Study 2, 3), and benefits such as monetary compensation (Study 4) affect overreliance. Finally, Study 5 adapts the Cognitive Effort Discounting paradigm to quantify the utility of different explanations, providing further support for our framework. Our results suggest that some of the null effects found in literature could be due in part to the explanation not sufficiently reducing the costs of verifying the AI's prediction.

Motivation & Objective

• Motivate and test whether explanations can mitigate overreliance by framing engagement with AI as a cost-benefit decision.
• Formalize a cost-benefit framework where cognitive effort, task difficulty, and incentives influence whether users engage with an AI explanation.
• Investigate how varying task difficulty, explanation complexity, and monetary incentives affect reliance on AI predictions and explanations.
• Demonstrate that explanations can reduce overreliance under certain conditions, challenging the view that explanations never help.
• Quantify the utility of explanations using a cognitive effort discounting paradigm to link costs, benefits, and explanatory usefulness.

Proposed method

• Develop a cost-benefit framework where users choose between engaging with the task (with or without verifying the AI) or relying on the AI.
• Use a maze-solving task with a simulated AI to manipulate task difficulty, explanation difficulty, and monetary incentives across five studies.
• Employ two explanation modalities (highlight and written explanations) and ensure AI accuracy is balanced at 80% across studies.
• Generate accurate explanations for correct AI predictions and misleading paths when AI errs to study their effect on verification behavior.
• Measure overreliance, task engagement, and subjective utility, supplemented by self-report scales and a Cognitive Effort Discounting approach in Study 5.

Experimental results

Research questions

• RQ1 Under what conditions do people engage with AI explanations to verify predictions and reduce overreliance?
• RQ2 How do task difficulty, explanation difficulty, and monetary incentives shape the utility of engaging with explanations versus relying on AI?
• RQ3 Can a cost-benefit perspective account for when explanations decrease overreliance rather than fail to do so?

Key findings

• As task difficulty increases, explanations reduce overreliance more than predictions alone.
• Lower cognitive effort to understand explanations leads to greater reductions in overreliance.
• Higher monetary rewards for accuracy decrease overreliance, showing benefits influence engagement strategy.
• Explanations that are easier to understand receive higher subjective utility, increasing the likelihood of engaging with the task.
• The adapted Cognitive Effort Discounting paradigm confirms that explanations with lower understanding costs are valued more, and that task difficulty raises the utility of explanations.