[Paper Review] On the Relationship Between Explanations, Fairness Perceptions, and Decisions

Jakob Schoeffer, Maria De‐Arteaga+1 more|arXiv (Cornell University)|Apr 27, 2022

Ethics and Social Impacts of AI4 citations

TL;DR

This paper proposes a conceptual framework linking explanations, procedural fairness perceptions, reliance on AI, and distributive fairness in human-AI decision-making. It identifies that misleading explanations can distort fairness perceptions, leading to inappropriate reliance on AI and undermining distributive fairness—highlighting a critical gap in current XAI research.

ABSTRACT

It is known that recommendations of AI-based systems can be incorrect or unfair. Hence, it is often proposed that a human be the final decision-maker. Prior work has argued that explanations are an essential pathway to help human decision-makers enhance decision quality and mitigate bias, i.e., facilitate human-AI complementarity. For these benefits to materialize, explanations should enable humans to appropriately rely on AI recommendations and override the algorithmic recommendation when necessary to increase distributive fairness of decisions. The literature, however, does not provide conclusive empirical evidence as to whether explanations enable such complementarity in practice. In this work, we (a) provide a conceptual framework to articulate the relationships between explanations, fairness perceptions, reliance, and distributive fairness, (b) apply it to understand (seemingly) contradictory research findings at the intersection of explanations and fairness, and (c) derive cohesive implications for the formulation of research questions and the design of experiments.

Motivation & Objective

• To address the lack of empirical evidence on whether explanations enable human-AI complementarity in fair decision-making.
• To resolve seemingly contradictory findings in XAI and fairness research by identifying the mediating role of procedural fairness perceptions.
• To investigate how misleading explanations may distort perceptions and lead to incorrect reliance on AI, harming distributive fairness.
• To provide a holistic framework for evaluating explanations beyond fairness perceptions, including their impact on reliance and actual decision outcomes.

Proposed method

• Proposes a conceptual framework mapping the causal pathway from explanations to distributive fairness via procedural fairness perceptions and reliance on AI.
• Classifies prior research into three clusters: (A) XAI and fairness perceptions, (B) XAI and reliance, (C) reliance and distributive fairness.
• Uses the framework to analyze inconsistencies in prior findings, identifying where research is fragmented or incomplete.
• Identifies that explanations can mislead perceptions even without intent to manipulate, using examples like feature-based deception in attention explanations.
• Proposes that miscalibrated perceptions due to misleading explanations may lead to unwarranted AI adoption or inappropriate override.
• Outlines a future human subject study to empirically test the full pathway from explanations to distributive fairness via perception and reliance.

Experimental results

Research questions

• RQ1 Given that explanations can mislead perceptions of fairness, how does this mislead reliance on AI recommendations in ways detrimental to distributive fairness?
• RQ2 To what extent do misleading explanations lead to unwarranted trust in AI, even when recommendations are unfair or incorrect?
• RQ3 How do procedural fairness perceptions mediate the relationship between explanations and actual reliance behavior on AI advice?
• RQ4 In what ways can explanations that appear fair in form but are misleading in content result in unequal decision outcomes across demographic groups?

Key findings

• Explanations can mislead people into perceiving AI systems as procedurally fair even when they rely on sensitive attributes like race or gender.
• Misleading explanations have been shown to deceive users into trusting models that are not fair, using correlations between legitimate and sensitive features.
• Prior studies show that explanations do not consistently improve human decision-making or reduce bias, and in some cases lead to over-reliance on AI.
• There is no conclusive evidence that explanations enhance human-AI complementarity or improve distributive fairness in practice.
• Reliance on AI is influenced by fairness perceptions, and when perceptions are miscalibrated due to deceptive explanations, reliance can become inappropriate.
• The framework reveals a critical gap: most XAI research focuses on isolated components (e.g., perceptions or performance) rather than the full chain from explanation to fairness outcome.