[Paper Review] Towards Involving End-users in Interactive Human-in-the-loop AI Fairness

Yuri Nakao, Simone Stumpf+3 more|arXiv (Cornell University)|Apr 22, 2022

Ethics and Social Impacts of AI4 citations

TL;DR

This paper proposes an interactive, human-in-the-loop AI fairness interface enabling non-technical end-users to identify and correct fairness issues in loan decision AI systems through explanatory debugging. By allowing users to adjust feature weights based on interpretability features like model confidence and attribute values, the system enables end-users to improve fairness, with cultural dimensions—particularly Masculinity, Uncertainty Avoidance, and Indulgence—significantly influencing fairness judgments.

ABSTRACT

Ensuring fairness in artificial intelligence (AI) is important to counteract bias and discrimination in far-reaching applications. Recent work has started to investigate how humans judge fairness and how to support machine learning (ML) experts in making their AI models fairer. Drawing inspiration from an Explainable AI (XAI) approach called \emph{explanatory debugging} used in interactive machine learning, our work explores designing interpretable and interactive human-in-the-loop interfaces that allow ordinary end-users without any technical or domain background to identify potential fairness issues and possibly fix them in the context of loan decisions. Through workshops with end-users, we co-designed and implemented a prototype system that allowed end-users to see why predictions were made, and then to change weights on features to "debug" fairness issues. We evaluated the use of this prototype system through an online study. To investigate the implications of diverse human values about fairness around the globe, we also explored how cultural dimensions might play a role in using this prototype. Our results contribute to the design of interfaces to allow end-users to be involved in judging and addressing AI fairness through a human-in-the-loop approach.

Motivation & Objective

• To design an interactive, interpretable human-in-the-loop interface that enables non-technical end-users to assess and correct fairness issues in AI-driven loan decisions.
• To investigate how end-users without technical or domain expertise perceive and act on fairness in AI systems using model explanations.
• To explore the role of cultural dimensions in shaping end-users' fairness judgments and interventions in AI systems.
• To evaluate the effectiveness of user-driven weight adjustments in improving model fairness through a prototype and online study.

Proposed method

• Adopted explanatory debugging from interactive machine learning to guide end-users in identifying and correcting fairness-related mispredictions.
• Designed a prototype system that visualizes model decisions using confidence scores, feature weights, decision boundaries, and similarity metrics.
• Enabled end-users to modify feature weights interactively to adjust model behavior and test fairness improvements.
• Conducted co-design workshops with end-users to inform interface design and ensure usability for non-experts.
• Employed Hofstede’s Cultural Dimensions framework to analyze cross-cultural differences in fairness perception and intervention behavior.
• Performed an online study with end-users to evaluate the prototype’s usability and impact on fairness perception and adjustment.

Experimental results

Research questions

• RQ1 How can interactive human-in-the-loop interfaces be designed to allow end-users without technical expertise to identify and correct fairness issues in AI systems?
• RQ2 How do end-users assess fairness in AI loan decisions, and what factors (e.g., confidence, weights, attribute values) influence their judgments?
• RQ3 To what extent do cultural dimensions—such as Masculinity, Uncertainty Avoidance, and Indulgence—affect end-users’ fairness assessments and interventions?

Key findings

• End-users effectively assessed fairness by analyzing model confidence, feature weights, and attribute values, using graphical comparisons of individual loan applications.
• The prototype enabled end-users to make meaningful fairness improvements by adjusting feature weights, demonstrating the feasibility of user-driven model correction.
• Cultural dimensions significantly influenced fairness perceptions, with Masculinity, Uncertainty Avoidance, and Indulgence emerging as key factors shaping user behavior and judgment.
• Users demonstrated understanding of model behavior when explanations were presented clearly, indicating that interpretability is essential for effective end-user involvement.
• The study revealed that end-users prioritized fairness based on perceived justice and risk, which varied across cultural contexts, highlighting the need for culturally aware AI design.
• Despite limitations in real-time feedback and model retraining, the offline prototype successfully demonstrated the potential of human-in-the-loop fairness interventions.