[Paper Review] Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences

Tim Miller, Piers D. L. Howe+1 more|arXiv (Cornell University)|Dec 2, 2017

Explainable Artificial Intelligence (XAI)25 references209 citations

TL;DR

The paper argues that explainable AI often reflects researchers’ own needs rather than end users, and advocates grounding XAI in social/behavioral science research with human-centered evaluation. It surveys related work and highlights how explanations are social, contrastive, and need human-in-the-loop validation.

ABSTRACT

In his seminal book `The Inmates are Running the Asylum: Why High-Tech Products Drive Us Crazy And How To Restore The Sanity' [2004, Sams Indianapolis, IN, USA], Alan Cooper argues that a major reason why software is often poorly designed (from a user perspective) is that programmers are in charge of design decisions, rather than interaction designers. As a result, programmers design software for themselves, rather than for their target audience, a phenomenon he refers to as the `inmates running the asylum'. This paper argues that explainable AI risks a similar fate. While the re-emergence of explainable AI is positive, this paper argues most of us as AI researchers are building explanatory agents for ourselves, rather than for the intended users. But explainable AI is more likely to succeed if researchers and practitioners understand, adopt, implement, and improve models from the vast and valuable bodies of research in philosophy, psychology, and cognitive science, and if evaluation of these models is focused more on people than on technology. From a light scan of literature, we demonstrate that there is considerable scope to infuse more results from the social and behavioural sciences into explainable AI, and present some key results from these fields that are relevant to explainable AI.

Motivation & Objective

• Argue that explainable AI risks being driven by researchers’ perspectives rather than users’ needs.
• Show that social sciences and human factors are underrepresented in XAI literature.
• Advocate incorporating explanatory models from philosophy, psychology, and cognitive science into XAI.
• Encourage evaluation of explanations using data from human behavior studies.

Proposed method

• Conduct a lightweight survey of 23 articles from the IJCAI 2017 XAI workshop Related Work list to assess exposure to social science explanations.
• Categorize papers by topic (on-topic vs off-topic) and by data-driven and validation criteria based on social science references and human-behavior data.
• Summarize key ideas from social science literatures relevant to explanation (contrastive explanation, attribution theory, explanation selection, evaluation, and conversational nature of explanations).
• Identify gaps and potential impact points for integrating social science insights into XAI model design and evaluation.

Experimental results

Research questions

• RQ1 To what extent do XAI papers build on social science theories of explanation?
• RQ2 How often are human behavioral experiments used to validate explanations in XAI literature?
• RQ3 What social science concepts (e.g., contrastive explanations, attribution, explanation selection) can inform XAI models?
• RQ4 What practical steps can researchers take to align XAI with human-centered evaluation standards?

Key findings

• Many XAI papers draw little from social sciences or human factors; only a subset references social science explanations, and even fewer ground their work in human behavioral data.
• Human behavioral experiments in evaluating explanations are rare in the surveyed literature.
• Contrastive explanations, attribution theory, and explanation selection from social sciences offer valuable guidance for designing and evaluating explanations in AI.
• Explanations should be treated as interactive conversations governed by conversational maxims, not merely as static disclosures of model internals.
• The paper argues for greater collaboration with social/behavioral scientists and for evaluations informed by human behavior data, aligning with DARPA’s human-in-the-loop emphasis.