[Paper Review] Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems

Richard Tomsett, Dave Braines+3 more|ORCA Online Research @Cardiff (Cardiff University)|Jun 20, 2018

Explainable Artificial Intelligence (XAI)Computer Science19 references84 citations

TL;DR

The paper proposes a role-based model to analyze interpretability in machine learning systems by identifying different agent roles and examining how these roles shape interpretability goals.

ABSTRACT

Several researchers have argued that a machine learning system's interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent's role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.

Motivation & Objective

• Motivate that interpretability is relative to a specific agent or task, not a universal property.
• Introduce a role-based model to identify agent roles related to a ML system.
• Illustrate how an agent's role shapes interpretability goals and requirements.
• Discuss implications for researchers, system developers, and regulatory auditing of ML systems.

Proposed method

• Define a role-based framework to categorize agents interacting with ML systems.
• Analyze scenarios to show how an agent’s role affects interpretability objectives.
• Discuss implications for defining interpretability across different stakeholder roles.

Experimental results

Research questions

• RQ1 What are the agent roles that interact with ML systems relevant to interpretability?
• RQ2 How do different agent roles influence interpretability goals and assessments?
• RQ3 What are the broader implications for research, development, and regulation of interpretable ML?

Key findings

• A role-based perspective helps explain why interpretability is not a single property but depends on the agent and context.
• Illustrative scenarios demonstrate how an agent’s goals alter what counts as interpretable.
• The model has implications for how interpretability is defined, evaluated, and audited across stakeholders.
• The authors provide guidance on how the model can be useful to interpretability researchers, system developers, and regulatory bodies.