[Paper Review] Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions

Upol Ehsan, Pradyumna Tambwekar+3 more|arXiv (Cornell University)|Jan 11, 2019

Explainable Artificial Intelligence (XAI)37 references44 citations

TL;DR

The paper trains neural rationale generators to produce human-like natural language explanations for an agent's actions in Frogger, comparing focused-view and complete-view configurations through two user studies on perceived quality and preferences.

ABSTRACT

Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.

Motivation & Objective

• Create a high-quality corpus of human explanations via remote think-aloud protocols tied to game states and actions.
• Train encoder-decoder neural models to translate state-action representations into natural language rationales.
• Evaluate how rationale style affects user perceptions across confidence, human-likeness, justification, and understandability.
• Compare focused-view versus complete-view rationale generators and analyze user preferences and alignment with design goals.

Proposed method

• Collect a large corpus of think-aloud explanations paired with game states using a modified Frogger interface.
• Train an encoder-decoder RNN with GRU cells and an attention mechanism to translate serialized state-action input into rationales.
• Experiment with two input configurations: focused-view (7x7 local window) and complete-view (full board with input noise).
• Use 100 training epochs for the rationale generator and evaluate on plausibility and perceptual dimensions via user studies.
• Conduct two user studies measuring confidence, human-likeness, adequate justification, and understandability, with comparisons to exemplars and random baselines.

Experimental results

Research questions

• RQ1 Do generated rationales (focused-view and complete-view) outperform random baselines in plausibility and perceived quality?
• RQ2 How do rationales differ in perceived confidence, human-likeness, justification adequacy, and understandability?
• RQ3 Does complete-view reasoning yield different user perceptions compared to focused-view reasoning?
• RQ4 Are users preferential toward one rationale style (focused-view vs complete-view) and why?

Key findings

• Both focused-view and complete-view rationales significantly outperform random baselines across all four perception dimensions.
• Exemplary human rationales are rated higher than candidate rationales, with differences varying by configuration.
• Complete-view candidate rationales generally receive higher ratings than focused-view candidate rationales.
• Rationales with contextual accuracy, awareness, and strategic detail drive higher confidence and understandability; intelligibility alone is insufficient for high confidence.
• Participants preferred rationales that provided detailed, holistic explanations when context permitted, aiding stable mental models.