[Paper Review] Sequential Explanations with Mental Model-Based Policies

Arnold YS Yeung, Shalmali Joshi+2 more|arXiv (Cornell University)|Jul 17, 2020

Explainable Artificial Intelligence (XAI)31 references4 citations

TL;DR

This paper proposes a reinforcement learning framework that generates sequential explanations tailored to a user's evolving mental model, improving interpretability over time. By observing users' comprehension through behavioral proxies, the method selects explanations that increase simulatability more effectively than random selection, especially with prototype and combined explanation types.

ABSTRACT

The act of explaining across two parties is a feedback loop, where one provides information on what needs to be explained and the other provides an explanation relevant to this information. We apply a reinforcement learning framework which emulates this format by providing explanations based on the explainee's current mental model. We conduct novel online human experiments where explanations generated by various explanation methods are selected and presented to participants, using policies which observe participants' mental models, in order to optimize an interpretability proxy. Our results suggest that mental model-based policies (anchored in our proposed state representation) may increase interpretability over multiple sequential explanations, when compared to a random selection baseline. This work provides insight into how to select explanations which increase relevant information for users, and into conducting human-grounded experimentation to understand interpretability.

Motivation & Objective

• To address the gap in user-centered explanation design by modeling explanation as a feedback loop between explainer and explainee.
• To improve interpretability of black-box models by dynamically selecting explanations based on the explainee’s current mental model.
• To evaluate whether mental model-based policies outperform random selection in enhancing user simulatability over sequential explanations.
• To develop and deploy online human experiments using MTurk to ground interpretability metrics in real user behavior.
• To explore the feasibility of using empirical mental model observations as state representations in an RL framework for explanation selection.

Proposed method

• The framework models explanation as a sequential decision-making process using reinforcement learning, where the agent selects explanations based on observed user mental models.
• Mental models are represented via empirical behavioral proxies—such as user responses and interaction patterns—measured during online human experiments.
• The policy selects explanations from multiple types (e.g., saliency maps, prototypes, combined) based on their expected impact on interpretability, using a simulatability proxy as the reward signal.
• Explanations are generated using established explainers: deep Taylor decomposition for saliency maps and prototype-based methods for local representation.
• Online experiments were conducted on Amazon MTurk, where participants interacted with explanations in sequence, and their comprehension was measured via simulatability scores.
• Cohen’s d effect size was computed to assess the magnitude of improvement in simulatability between mental model-based policies and random baselines.

Experimental results

Research questions

• RQ1 Can a reinforcement learning policy that adapts to a user’s evolving mental model improve interpretability over sequential explanations?
• RQ2 How does the performance of mental model-based explanation policies compare to random selection in terms of user simulatability?
• RQ3 Do different explanation types (e.g., saliency maps vs. prototypes) vary in their effectiveness when selected based on mental models?
• RQ4 To what extent do mental model observations reduce irrelevant information and increase comprehension over time?
• RQ5 What is the impact of explanation diversity and complementarity on long-term interpretability gains?

Key findings

• Mental model-based policies significantly increased simulatability scores compared to random selection, particularly for prototype and combined explanation types.
• The effect size (Cohen’s d) for mental model-based policies was medium to large (d > 0.5), indicating a meaningful improvement in interpretability.
• Saliency map-based policies showed similar performance between mental model and random baselines, suggesting limited gains for this explanation type.
• The plateauing of simulatability scores in saliency and prototype experiments may indicate diminishing returns or cognitive overload after multiple explanations.
• The absence of plateauing in combined explanations suggests that diverse explanation types provide increasingly complementary information, sustaining interpretability gains.
• High standard errors in simulatability scores indicate variability in user responses, likely due to fatigue, attention shifts, or individual differences not monitored in online experiments.