[Paper Review] Fairness of ChatGPT

Yunqi Li, Yongfeng Zhang+1 more|arXiv (Cornell University)|May 22, 2023

Ethics and Social Impacts of AI11 citations

TL;DR

The paper systematically evaluates the fairness of ChatGPT in high-stakes domains (education, criminology, finance, and healthcare), analyzing both group and individual fairness under biased and unbiased prompts, and comparing with small baselines.

ABSTRACT

Understanding and addressing unfairness in LLMs are crucial for responsible AI deployment. However, there is a limited number of quantitative analyses and in-depth studies regarding fairness evaluations in LLMs, especially when applying LLMs to high-stakes fields. This work aims to fill this gap by providing a systematic evaluation of the effectiveness and fairness of LLMs using ChatGPT as a study case. We focus on assessing ChatGPT's performance in high-takes fields including education, criminology, finance and healthcare. To conduct a thorough evaluation, we consider both group fairness and individual fairness metrics. We also observe the disparities in ChatGPT's outputs under a set of biased or unbiased prompts. This work contributes to a deeper understanding of LLMs' fairness performance, facilitates bias mitigation and fosters the development of responsible AI systems.

Motivation & Objective

• Assess ChatGPT's fairness and effectiveness in high-stakes domains using four datasets (education, criminology, finance, healthcare).
• Evaluate both group fairness (e.g., equal opportunity, demographic parity) and individual fairness (counterfactual fairness).
• Investigate model sensitivity to biased vs. unbiased in-context prompts and their impact on outputs.
• Compare ChatGPT with small baseline models (logistic regression, MLP) on identical tasks.
• Provide data, prompts, and results to support bias mitigation and responsible AI deployment.

Proposed method

• Use four standard fairness datasets: PISA (education), COMPAS (criminology), German Credit (finance), Heart Disease (healthcare).
• Train small baselines (logistic regression, MLP) and compare with ChatGPT (gpt-3.5-turbo) under zero-temperature prompts.
• Design eight prompts (four unbiased, four biased) with in-context examples to test prompt sensitivity.
• Evaluate with accuracy, F1, AUC (effectiveness) and group-level (SP, TPR, FPR, ACC, F1, AUC) and counterfactual (CR) fairness metrics.
• Construct counterfactual testing sets by flipping the sensitive feature to assess individual fairness.

Experimental results

Research questions

• RQ1 How does ChatGPT perform on high-stakes tasks compared with small models in terms of accuracy and fairness?
• RQ2 What are the group fairness behaviors (e.g., SP, TPR, FPR) across gender and race groups when using ChatGPT?
• RQ3 How does ChatGPT respond to biased versus unbiased prompts in terms of outputs and fairness metrics?
• RQ4 To what extent does ChatGPT exhibit counterfactual fairness on these domains?

Key findings

• ChatGPT can match small models in overall effectiveness on PISA, COMPAS, and Heart Disease, but underperforms on German Credit.
• ChatGPT generally achieves better group fairness than small models but shows notable individual fairness gaps across datasets.
• Biased prompts often lead to worse fairness outcomes; unbiased prompts tend to yield better performance, though trends vary by dataset.
• Counterfactual examples in prompts do not consistently improve fairness and sometimes reduce effectiveness.
• Prompt design substantially influences outputs, highlighting the need for careful prompt engineering to mitigate bias.