[Paper Review] BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models

Rafał Kocielnik, Shrimai Prabhumoye+4 more|arXiv (Cornell University)|Feb 14, 2023

Topic Modeling4 citations

TL;DR

This paper introduces BiasTestGPT, a framework that leverages ChatGPT to automatically generate diverse, naturalistic test sentences for detecting social biases in pretrained language models (PLMs). By enabling on-demand, flexible bias specification and integrating with HuggingFace, the tool significantly improves detection of intersectional and subtle biases compared to template-based or crowd-sourced methods, offering domain experts an accessible, scalable, and open-source solution for bias testing.

ABSTRACT

Pretrained Language Models (PLMs) harbor inherent social biases that can result in harmful real-world implications. Such social biases are measured through the probability values that PLMs output for different social groups and attributes appearing in a set of test sentences. However, bias testing is currently cumbersome since the test sentences are generated either from a limited set of manual templates or need expensive crowd-sourcing. We instead propose using ChatGPT for the controllable generation of test sentences, given any arbitrary user-specified combination of social groups and attributes appearing in the test sentences. When compared to template-based methods, our approach using ChatGPT for test sentence generation is superior in detecting social bias, especially in challenging settings such as intersectional biases. We present an open-source comprehensive bias testing framework (BiasTestGPT), hosted on HuggingFace, that can be plugged into any open-source PLM for bias testing. User testing with domain experts from various fields has shown their interest in being able to test modern AI for social biases. Our tool has significantly improved their awareness of such biases in PLMs, proving to be learnable and user-friendly. We thus enable seamless open-ended social bias testing of PLMs by domain experts through an automatic large-scale generation of diverse test sentences for any combination of social categories and attributes.

Motivation & Objective

• To address the limitations of existing social bias testing methods, which rely on small, artificial, or expensive crowd-sourced test sets.
• To enable domain experts to systematically test for social biases in PLMs using customizable, contextually rich test sentences.
• To reduce reliance on manual template creation or costly human annotation by automating test sentence generation via large language models.
• To improve detection of complex, intersectional biases that are often missed by traditional methods.
• To provide an open-source, user-friendly tool integrated with HuggingFace for seamless bias testing across any open-source PLM.

Proposed method

• The framework uses ChatGPT to generate natural, contextually diverse test sentences based on user-specified combinations of social groups and attributes.
• It generates paired sentence variants—stereotype and anti-stereotype—to enable comparative probability analysis for bias quantification.
• The system supports flexible bias specification through term-based inputs, allowing users to define custom social categories and attributes.
• Test sentences are grouped into stereotype/anti-stereotype pairs and evaluated using a bias metric from Nadeem et al. (2021) to quantify bias levels.
• The tool is hosted on HuggingFace, enabling plug-and-play integration with any HuggingFace-hosted PLM for real-time bias evaluation.
• The framework supports multi-level output: model-level, attribute-level, and sentence-level bias analysis for granular inspection.

Figure 1. An overview of the Graphical User Interface of our open-source HuggingFace tool for social bias testing in PLMs. The tool connects to ChatGPT and supports step-by-step bias testing workflow (A). Following a flexible term-based bias specification by domain expert (B) the tool can retrieve o

Experimental results

Research questions

• RQ1 Can LLM-generated test sentences outperform template-based or crowd-sourced methods in detecting social biases in PLMs, especially in complex, intersectional settings?
• RQ2 To what extent can ChatGPT be used to generate diverse, contextually natural, and representative test sentences for bias testing?
• RQ3 How usable and effective is the BiasTestGPT framework for domain experts without NLP expertise in identifying and measuring social biases in PLMs?
• RQ4 Does the automated generation of test sentences improve the detection of subtle or intersectional biases compared to traditional methods?
• RQ5 How does the framework’s performance compare to existing intrinsic bias evaluation methods in terms of reliability and scalability?

Key findings

• BiasTestGPT significantly improves the detection of social bias in PLMs, particularly in challenging intersectional bias scenarios, compared to template-based methods.
• The framework enables domain experts to generate high-quality, contextually diverse test sentences on demand, increasing their awareness of fairness issues in AI.
• User testing with domain experts demonstrated strong usability and interest in the tool, confirming its learnability and practical value.
• The generated test sets are automatically open-sourced in HuggingFace format, ensuring immediate accessibility and reproducibility.
• The tool supports fine-grained, sentence-level inspection, allowing users to manually verify and validate bias findings.
• Despite limitations related to the training data distribution of the underlying LLM, the framework provides a scalable and flexible alternative to costly or rigid bias testing pipelines.

Figure 2. Overview of our BiasTestGPT framework for test sentence generation for social bias testing in pre-trained language models. We leverage ChatGPT to generate sentences to test social bias on a Tested PLM . The steps involved: (1) user-provided social bias specification; (2) ChatGPT generation