[Paper Review] How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation

Menaka Narayanan, Emily Chen+4 more|arXiv (Cornell University)|Feb 2, 2018

Explainable Artificial Intelligence (XAI)42 references97 citations

TL;DR

The paper empirically investigates which properties of decision-set explanations most affect humans’ ability to verify outputs against inputs, using two domains (recipe and clinical) and multiple complexity factors.

ABSTRACT

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable in the specific context of verification. Suppose we have a machine learning system that predicts X, and we provide rationale for this prediction X. Given an input, an explanation, and an output, is the output consistent with the input and the supposed rationale? Via a series of user-studies, we identify what kinds of increases in complexity have the greatest effect on the time it takes for humans to verify the rationale, and which seem relatively insensitive.

Motivation & Objective

• Quantify what makes explanations human-interpretable in the verification task.
• Identify which factors of decision-set explanations most increase verification effort.
• Assess whether domain context (recipe vs clinical) affects explanation processing.
• Provide guidance for designing human-friendly explanations in ML systems.

Proposed method

• Use controlled user studies with synthetic explanation selections presented as decision sets.
• Manipulate explanation size by varying the number of lines and output-term length.
• Introduce new cognitive chunks and test explicit vs implicit chunking.
• Vary repetition of input terms across lines to measure search effort.
• Test in two domains (alien recipe recommendations and alien medical treatments) with parallel tasks.
• Measure response time, accuracy, and subjective satisfaction for each condition.

Experimental results

Research questions

• RQ1 What explanation properties (size, cognitive chunking, and term repetition) most affect humans’ verification performance?
• RQ2 Do explicit introduction of new concepts vs embedding them implicitly affect processing time and satisfaction?
• RQ3 Are effects of explanation complexity consistent across different domains (recipe vs clinical)?
• RQ4 How do explanation complexity factors influence accuracy and subjective trust in the explanations?

Key findings

• Increasing explanation complexity generally raises response time and lowers satisfaction.
• The number of lines and the length of output clauses most strongly increase processing time.
• Introducing new cognitive chunks (explicit) tends to increase processing time more than embedding concepts implicitly, and can reduce satisfaction.
• Repeated terms have a subtler impact on response time and satisfaction compared to adding lines or new concepts.
• Accuracy is relatively robust to variation in explanation complexity, while processing cost shifts primarily to response time and satisfaction.
• Results are broadly consistent across the recipe and clinical domains, suggesting generalizable principles for explanation design.