[Paper Review] How Do Fairness Definitions Fare? Examining Public Attitudes Towards Algorithmic Definitions of Fairness

Nripsuta Ani Saxena, Karen Huang+4 more|arXiv (Cornell University)|Nov 8, 2018

Ethics and Social Impacts of AI23 references49 citations

TL;DR

The paper investigates ordinary people's perceptions of three algorithmic fairness definitions in loan decisions, using two online experiments to see how perceptions change when race information is added, with calibrated fairness often favored and affirmative action supported.

ABSTRACT

What is the best way to define algorithmic fairness? While many definitions of fairness have been proposed in the computer science literature, there is no clear agreement over a particular definition. In this work, we investigate ordinary people's perceptions of three of these fairness definitions. Across two online experiments, we test which definitions people perceive to be the fairest in the context of loan decisions, and whether fairness perceptions change with the addition of sensitive information (i.e., race of the loan applicants). Overall, one definition (calibrated fairness) tends to be more preferred than the others, and the results also provide support for the principle of affirmative action.

Motivation & Objective

• Understand which CS fairness definitions align with lay perceptions in lending contexts.
• Assess how adding sensitive attributes (race) affects fairness judgments.
• Compare public support for different fairness definitions.
• Explore implications for deploying fairness notions in real-world decision systems.

Proposed method

• Conduct two online experiments presenting loan decision scenarios to participants.
• Manipulate which fairness definition is applied in each scenario.
• Vary the inclusion of sensitive information (applicant race) to test its impact on fairness judgments.
• Measure perceived fairness and preferences across definitions.
• Analyze whether responses differ by exposure to race information and other factors.

Experimental results

Research questions

• RQ1 Which algorithmic fairness definitions are perceived as fairest by ordinary people in loan decision contexts?
• RQ2 Does the inclusion of sensitive information (race) influence fairness judgments across definitions?
• RQ3 Do participants vary in preference among calibrated fairness, statistical parity, and other definitions when race is present or absent?
• RQ4 Is there evidence supporting affirmative action as a fairness approach in public perception?

Key findings

• Calibrated fairness tends to be more preferred than the other definitions examined.
• Public attitudes show support for affirmative action principles in the context studied.
• Adding sensitive information (race) affects fairness judgments, influencing preferences among definitions.
• The results illuminate which fairness notions align with lay intuitions in lending scenarios.