[Paper Review] To Explain or Not to Explain: A Study on the Necessity of Explanations for Autonomous Vehicles

Yuan Shen, Shanduojiao Jiang+2 more|arXiv (Cornell University)|Jun 21, 2020

Explainable Artificial Intelligence (XAI)Computer Science26 references31 citations

TL;DR

The paper investigates when explanations for autonomous vehicle actions are needed and how driver type and driving context influence explanation necessity, introducing a dataset of 1103 video clips with labeled explanation necessity.

ABSTRACT

Explainable AI, in the context of autonomous systems, like self-driving cars, has drawn broad interests from researchers. Recent studies have found that providing explanations for autonomous vehicles' actions has many benefits (e.g., increased trust and acceptance), but put little emphasis on when an explanation is needed and how the content of explanation changes with driving context. In this work, we investigate which scenarios people need explanations and how the critical degree of explanation shifts with situations and driver types. Through a user experiment, we ask participants to evaluate how necessary an explanation is and measure the impact on their trust in self-driving cars in different contexts. Moreover, we present a self-driving explanation dataset with first-person explanations and associated measures of the necessity for 1103 video clips, augmenting the Berkeley Deep Drive Attention dataset. Our research reveals that driver types and driving scenarios dictate whether an explanation is necessary. In particular, people tend to agree on the necessity for near-crash events but hold different opinions on ordinary or anomalous driving situations.

Motivation & Objective

• Assess how driving scenarios influence the necessity of text-based explanations for self-driving decisions.
• Examine how driver types (e.g., aggressive vs. cautious) affect explanation needs across contexts.
• Identify whether a universal explanation content format exists across driving scenarios.
• Create a dataset annotating explanation necessity, timing, and content for autonomous driving videos.

Proposed method

• Conduct an online survey-based user study with 18 participants acting as vehicle passengers across 38 driving scenarios derived from clustering explanation annotations.
• Collect necessity scores, attentiveness, and preferred explanation content after each video clip.
• Define and compute a 'critical score' (0 to 1) indicating explanation necessity for each moment.
• Augment the Berkeley Deep Drive Attention dataset to create 1103 video clips annotated with explanation moments and necessity scores.
• Develop a spatial-temporal recurrent model to infer explanation necessity from video frames and compare to random guessing.

Experimental results

Research questions

• RQ1 Does explanation necessity correlate with driving scenarios and driver types?
• RQ2 Is there a generally preferred explanation content format across scenarios?
• RQ3 Does the presence of explanations increase user trust in autonomous vehicles?
• RQ4 What are the temporal dynamics of explanation necessity in near-crash versus ordinary driving events?
• RQ5 Can a dataset of first-person explanations and necessity scores support real-time inference of explanation needs?

Key findings

• Explanation necessity is correlated with both driver type and driving scenario, with near-crash situations showing higher necessity.
• There is substantial disagreement among participants on explanation necessity for ordinary or anomalous driving situations.
• A general, globally preferred explanation format across all scenarios was not found. For 16 of 38 scenarios, a preferred format could be identified, but no consistent pattern emerged.
• Aggressive drivers reported lower explanation necessity on average than cautious drivers (about 18% lower).
• A self-driving explanation dataset with 1103 clips shows varying explanation moments and necessity scores (0 to 1) and enables a baseline AUC of 0.6295–0.6794 for predicting explanation need.