[Paper Review] What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations

Michał Kuźba, Przemysław Biecek|arXiv (Cornell University)|Feb 7, 2020

Explainable Artificial Intelligence (XAI)Computer Science43 references21 citations

TL;DR

This paper proposes a conversational AI system, dr_ant, that collects user questions about a machine learning model's predictions to identify real-world needs for model explanations. By analyzing over 1,000 dialogues with a Titanic survival model, it reveals recurring user question patterns, offering a novel, interactive method to uncover end-user explanatory needs—shifting XAI development from developer-centric to human-centered design.

ABSTRACT

Recently we see a rising number of methods in the field of eXplainable Artificial Intelligence. To our surprise, their development is driven by model developers rather than a study of needs for human end users. The analysis of needs, if done, takes the form of an A/B test rather than a study of open questions. To answer the question "What would a human operator like to ask the ML model?" we propose a conversational system explaining decisions of the predictive model. In this experiment, we developed a chatbot called dr_ant to talk about machine learning model trained to predict survival odds on Titanic. People can talk with dr_ant about different aspects of the model to understand the rationale behind its predictions. Having collected a corpus of 1000+ dialogues, we analyse the most common types of questions that users would like to ask. To our knowledge, it is the first study which uses a conversational system to collect the needs of human operators from the interactive and iterative dialogue explorations of a predictive model.

Motivation & Objective

• To identify the actual explanatory needs of human users interacting with machine learning models, moving beyond developer-driven explanation methods.
• To investigate how open-ended, interactive dialogue with an ML model can reveal user expectations and questions about model behavior.
• To develop and deploy a conversational AI agent (dr_ant) that enables users to explore and question a black-box model in natural language.
• To analyze collected dialogues to extract recurring question types and user motivations for explanation, informing future XAI system design.
• To demonstrate that conversational interaction is a scalable and effective method for eliciting user needs in explainable AI, especially for non-expert users.

Proposed method

• Developed a chatbot named dr_ant to interact with users about predictions from a random forest model trained on the Titanic dataset.
• Implemented a multi-turn dialogue system with intent classification and entity recognition to handle user queries about passenger features, survival predictions, and explanations.
• Integrated model explanation tools (Ceteris Paribus profiles and Break Down plots) from the DALEX and Explanatory Model Analysis (EMA) frameworks to generate visual and textual explanations.
• Used a dynamic retraining pipeline: collected real user dialogues, added new intents, and expanded training data to improve system robustness and coverage.
• Deployed the system via web and Slack interfaces to collect 1,000+ real user dialogues in a naturalistic setting.
• Applied iterative dialogue management to support open-ended, user-initiated conversations, enabling exploration of unanticipated questions.

Experimental results

Research questions

• RQ1 What kinds of questions do human users actually ask when trying to understand a machine learning model’s predictions?
• RQ2 How can a conversational AI system effectively elicit and respond to diverse, open-ended user questions about model behavior?
• RQ3 What recurring patterns or themes emerge in user questions about model explanations, and how do they vary by user background or intent?
• RQ4 Can interactive, dialogue-based systems serve as a scalable method to discover and validate user needs for explainable AI?
• RQ5 How do user questions differ across roles (e.g., domain experts vs. lay users), and what does this reveal about their explanatory needs?

Key findings

• The most frequent user questions revolved around 'what-if' scenarios (e.g., 'What if the passenger were older?'), feature importance, and model fairness, especially regarding gender and age.
• Users frequently inquired about model behavior on specific individuals, indicating a strong need for instance-level explanations and counterfactual reasoning.
• A significant number of questions focused on fairness and bias, particularly concerning gender and age, revealing user concerns about discriminatory outcomes.
• The system successfully collected 1,000+ dialogues, demonstrating that conversational AI is a viable and scalable method for gathering user needs in XAI.
• The analysis revealed distinct question clusters: feature influence, prediction sensitivity, model behavior on specific cases, and fairness concerns—each requiring tailored explanation strategies.
• The conversational approach uncovered previously unanticipated user needs, such as requests for model limitations and uncertainty, which are often overlooked in static explanation methods.