[Paper Review] AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?

Ehsan Latif, Xiaoming Zhaı+1 more|arXiv (Cornell University)|Dec 17, 2023

Ethics and Social Impacts of AI4 citations

TL;DR

This study investigates whether training data balance affects gender bias, disparities, and fairness in AI-based automatic scoring of student essays. Using fine-tuned BERT and GPT-3.5 on mixed-gender, male-only, and female-only datasets, it finds that mixed-trained models show no significant scoring bias, lower mean score gaps (MSG), and higher fairness (Equalized Odds) than gender-specific models—indicating that balanced training reduces gender disparities and enhances fairness.

ABSTRACT

This study delves into the pervasive issue of gender issues in artificial intelligence (AI), specifically within automatic scoring systems for student-written responses. The primary objective is to investigate the presence of gender biases, disparities, and fairness in generally targeted training samples with mixed-gender datasets in AI scoring outcomes. Utilizing a fine-tuned version of BERT and GPT-3.5, this research analyzes more than 1000 human-graded student responses from male and female participants across six assessment items. The study employs three distinct techniques for bias analysis: Scoring accuracy difference to evaluate bias, mean score gaps by gender (MSG) to evaluate disparity, and Equalized Odds (EO) to evaluate fairness. The results indicate that scoring accuracy for mixed-trained models shows an insignificant difference from either male- or female-trained models, suggesting no significant scoring bias. Consistently with both BERT and GPT-3.5, we found that mixed-trained models generated fewer MSG and non-disparate predictions compared to humans. In contrast, compared to humans, gender-specifically trained models yielded larger MSG, indicating that unbalanced training data may create algorithmic models to enlarge gender disparities. The EO analysis suggests that mixed-trained models generated more fairness outcomes compared with gender-specifically trained models. Collectively, the findings suggest that gender-unbalanced data do not necessarily generate scoring bias but can enlarge gender disparities and reduce scoring fairness.

Motivation & Objective

• To examine whether gender imbalance in training data leads to bias, disparities, or unfairness in AI-based automatic scoring systems.
• To compare the performance of mixed-gender-trained models against gender-specific models in terms of scoring accuracy, mean score gaps, and fairness.
• To assess whether balanced training data can mitigate gender disparities and improve algorithmic fairness in educational AI applications.
• To challenge the assumption that AI inevitably reproduces societal gender biases, particularly in automated assessment contexts.

Proposed method

• Fine-tuned BERT and GPT-3.5 on three distinct training data splits: mixed-gender, male-only, and female-only datasets.
• Evaluated models using three bias analysis techniques: scoring accuracy difference (paired t-test), mean score gap (MSG) between human and machine scores by gender, and Equalized Odds (EO) for fairness.
• Trained models on over 1,000 human-graded student responses from male and female participants across six assessment items.
• Applied statistical analysis to compare model predictions against human-graded scores, focusing on gender-based differences in accuracy, score gaps, and prediction parity.
• Used EO to measure the equality of true positive and false positive rates across genders, with lower values indicating higher fairness.
• Conducted comparative analysis between mixed-trained and gender-specific models on a shared mixed-gender test set to isolate the impact of training data composition.

Experimental results

Research questions

• RQ1 Does training data imbalance lead to measurable gender bias in AI-based automatic scoring systems?
• RQ2 How do mixed-gender-trained models compare to gender-specific models in terms of mean score gaps (MSG) and fairness?
• RQ3 To what extent does model fairness, as measured by Equalized Odds (EO), vary between mixed and single-gender trained models?
• RQ4 Can balanced training data reduce or eliminate gender disparities in AI-generated scoring outcomes?
• RQ5 Is the presence of gender bias in AI scoring systems primarily driven by data imbalance or model architecture?

Key findings

• Mixed-trained models showed no significant scoring accuracy difference between male and female responses, indicating negligible gender bias.
• Mixed-trained models produced lower mean score gaps (MSG) compared to human-graded scores, suggesting reduced gender disparity compared to gender-specific models.
• The Equalized Odds (EO) values for mixed-trained models were consistently lower (e.g., 0.061 for GPT-3.5 and 0.067 for BERT) than for gender-specific models, indicating higher fairness.
• Gender-specific models (e.g., male-only or female-only trained) exhibited higher MSG and EO values, indicating increased disparity and reduced fairness.
• For BERT, the EO for the male-only model was 0.107 and for the female-only model was 0.074, both higher than the mixed model’s 0.067.
• For GPT-3.5, the EO for the mixed model was 0.061, while the male-only and female-only models had EO values of 0.076 and 0.074, respectively, confirming the fairness advantage of mixed training.