[Paper Review] GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models

Tao Zhang, Ziqian Zeng+5 more|arXiv (Cornell University)|Jun 20, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces GenderAlign, a publicly available 8,000-sample alignment dataset designed to reduce gender bias in large language models (LLMs). Using GPT-3.5 to generate paired 'chosen' (low-bias, high-quality) and 'rejected' (high-bias) responses based on seed texts from existing datasets and books, the authors demonstrate that fine-tuning LLMs on GenderAlign significantly reduces gender bias compared to existing datasets like HH-RLHF, with human and LLM evaluators consistently preferring outputs from GenderAlign-trained models.

ABSTRACT

Large Language Models (LLMs) are prone to generating content that exhibits gender biases, raising significant ethical concerns. Alignment, the process of fine-tuning LLMs to better align with desired behaviors, is recognized as an effective approach to mitigate gender biases. Although proprietary LLMs have made significant strides in mitigating gender bias, their alignment datasets are not publicly available. The commonly used and publicly available alignment dataset, HH-RLHF, still exhibits gender bias to some extent. There is a lack of publicly available alignment datasets specifically designed to address gender bias. Hence, we developed a new dataset named GenderAlign, aiming at mitigating a comprehensive set of gender biases in LLMs. This dataset comprises 8k single-turn dialogues, each paired with a "chosen" and a "rejected" response. Compared to the "rejected" responses, the "chosen" responses demonstrate lower levels of gender bias and higher quality. Furthermore, we categorized the gender biases in the "rejected" responses of GenderAlign into 4 principal categories. The experimental results show the effectiveness of GenderAlign in reducing gender bias in LLMs.

Motivation & Objective

• To address the lack of publicly available alignment datasets specifically targeting gender bias mitigation in LLMs.
• To create a high-quality, diverse, and systematically categorized dataset of gender-biased and de-biased responses for fine-tuning LLMs.
• To evaluate the effectiveness of the proposed dataset in reducing gender bias across multiple LLMs and benchmarks.
• To analyze the contribution of different data sources (e.g., books, existing datasets) to bias mitigation performance.
• To provide a reproducible, automated pipeline for generating alignment data focused on gender equity.

Proposed method

• The dataset is generated using GPT-3.5 to produce 8,000 single-turn dialogues, each with a 'chosen' response (gender-unbiased, high-quality) and a 'rejected' response (gender-biased).
• Seed texts for gender-related topics are collected from two existing datasets (CORGI-PM, Workplace-Sexism) and five books on gender and gender roles.
• The 'chosen' response is generated by prompting GPT-3.5 to produce a response aligned with gender equity principles based on the dialogue context.
• The 'rejected' response is generated by removing the 'chosen' response from the context and prompting an unaligned LLM to generate a response, increasing the likelihood of bias.
• Gender biases in the 'rejected' responses are categorized into four main types: gender stereotypes, discriminatory language, sexism in institutions, and bias against marginalized genders.
• The dataset is evaluated using both LLM judges (GPT-3.5, Gemini-Pro, Claude-3-opus) and human evaluators, with preference rankings used to assess bias reduction.

Figure 1: “Chosen” and “ Rejected” response generation workflow. The input is a text either exhibits gender bias or describes gender difference.

Experimental results

Research questions

• RQ1 Can an automated, LLM-generated alignment dataset effectively reduce gender bias in fine-tuned LLMs compared to existing public datasets like HH-RLHF?
• RQ2 How do different data sources—such as books and existing datasets—contribute to the effectiveness of gender bias mitigation in the GenderAlign dataset?
• RQ3 To what extent does fine-tuning on GenderAlign reduce gender bias on standard benchmark datasets like BBQ and WinoGender?
• RQ4 How do human and LLM judges compare in their preference for responses from models fine-tuned on GenderAlign versus other alignment datasets?
• RQ5 Does the use of automated annotation via GPT-3.5 introduce new biases that undermine the dataset’s de-biasing goal?

Key findings

• Models fine-tuned on GenderAlign were preferred by human evaluators 51.1% of the time and by Claude-3-opus 65.8% of the time, significantly outperforming models trained on HH-RLHF and the Harmless dataset.
• The Llama2-13B model fine-tuned on GenderAlign achieved the lowest gender bias while maintaining high accuracy, outperforming both the base model and the Harmless-fine-tuned model on the BBQ benchmark.
• The average Pearson correlation coefficient for gender bias in profession-related stereotypes was lowest for models trained on GenderAlign, indicating reduced alignment with stereotypical gender-profession associations.
• Eliminating either the book-based or dataset-based components of GenderAlign led to decreased bias mitigation performance, with the Workplace-Sexism (CW) subset contributing more effectively than the books.
• Human evaluators showed substantial agreement (Fleiss’ Kappa = 0.655), indicating consistent preference for GenderAlign-trained model outputs across judges.
• The Harmless dataset, despite being designed for alignment, failed to reduce gender bias and even increased it in some cases, highlighting the limitations of existing alignment data for bias mitigation.

(a) GenderAlign dataset.