[Paper Review] Perceptions of Fairness and Trustworthiness Based on Explanations in Human vs. Automated Decision-Making

Jakob Schoeffer, Yvette Machowski+1 more|arXiv (Cornell University)|Sep 13, 2021

Ethics and Social Impacts of AI4 citations

TL;DR

This study investigates how people perceive fairness and trustworthiness in automated vs. human decision-making in high-stakes lending, using identical explanations for both. Surprisingly, participants perceived automated systems as significantly fairer than humans, especially those with high AI literacy, though no significant difference was found in perceived trustworthiness.

ABSTRACT

Automated decision systems (ADS) have become ubiquitous in many high-stakes domains. Those systems typically involve sophisticated yet opaque artificial intelligence (AI) techniques that seldom allow for full comprehension of their inner workings, particularly for affected individuals. As a result, ADS are prone to deficient oversight and calibration, which can lead to undesirable (e.g., unfair) outcomes. In this work, we conduct an online study with 200 participants to examine people's perceptions of fairness and trustworthiness towards ADS in comparison to a scenario where a human instead of an ADS makes a high-stakes decision -- and we provide thorough identical explanations regarding decisions in both cases. Surprisingly, we find that people perceive ADS as fairer than human decision-makers. Our analyses also suggest that people's AI literacy affects their perceptions, indicating that people with higher AI literacy favor ADS more strongly over human decision-makers, whereas low-AI-literacy people exhibit no significant differences in their perceptions.

Motivation & Objective

• To examine how people perceive fairness and trustworthiness in automated decision systems (ADS) versus human decision-makers in high-stakes contexts.
• To assess the impact of identical, thorough explanations on perceptions of fairness and trustworthiness in both ADS and human decision scenarios.
• To investigate how individual differences in AI literacy moderate perceptions of fairness and trustworthiness toward ADS versus human decision-makers.
• To explore the role of explainability in shaping public perceptions of automated systems in sensitive domains like lending.
• To contribute to information systems research by addressing the gap in understanding user perceptions of fairness and trust in algorithmic decision-making.

Proposed method

• Conducted an online experiment with 200 participants using a between-subjects design comparing ADS and human decision-makers in a lending context.
• Provided identical, detailed, model-agnostic explanations—specifically counterfactual explanations—for loan approval decisions in both conditions.
• Used a standardized scenario involving personal and financial data to ensure consistency across conditions.
• Measured perceptions of informational fairness and trustworthiness using validated survey scales.
• Collected and analyzed open-ended responses to gain qualitative insights into reasoning behind fairness and trust perceptions.
• Employed hierarchical linear modeling to test the moderating effect of AI literacy on perceptions of fairness and trustworthiness.

Experimental results

Research questions

• RQ1 How do people perceive the informational fairness of automated decision systems compared to human decision-makers when provided with identical explanations?
• RQ2 How do people perceive the trustworthiness of automated decision systems compared to human decision-makers under the same explanatory conditions?
• RQ3 Does AI literacy moderate the perception of fairness and trustworthiness toward automated decision systems relative to human decision-makers?
• RQ4 What qualitative factors influence participants’ perceptions of fairness and trust in automated versus human decision-making?

Key findings

• Participants perceived automated decision systems as significantly fairer than human decision-makers, with a p-value of 0.001 for the difference in informational fairness.
• No significant difference was found in perceived trustworthiness between automated and human decision-makers (p = 0.995).
• Individuals with high AI literacy perceived automated systems as both fairer and more trustworthy than human decision-makers, indicating a moderating effect of AI literacy.
• Participants with low AI literacy showed no significant difference in fairness or trustworthiness perceptions between the two conditions.
• Qualitative analysis revealed that participants viewed automated systems as fairer due to perceived objectivity and absence of subjectivity, with many stating that algorithms 'follow criteria' and 'use data objectively'.
• Some participants expressed concerns about accountability, noting that while the system is automated, 'someone decided to program it that way', highlighting a key ethical tension in ADS deployment.