[Paper Review] On Measuring Social Biases in Prompt-Based Multi-Task Learning

Afra Feyza Akyürek, Sejin Paik+4 more|arXiv (Cornell University)|May 23, 2022

Topic Modeling4 citations

TL;DR

This paper investigates how input formulation affects social biases in prompt-based multi-task learning, focusing on T0, a large-scale text-to-text language model. By comparing question-answer (QA) and premise-hypothesis (NLI) forms of semantically equivalent inputs, the authors find that T0 exhibits significantly higher bias in QA format—despite similar performance—due to training data alignment. They introduce BBNLI, the first bias benchmark for natural language inference with hand-written hypotheses, and release code and data to support future research on bias mitigation in multi-task models.

ABSTRACT

Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts, can generalize into novel forms of language and handle novel tasks. A large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. We consider an alternative measure and inquire whether the way in which an input is encoded affects social biases promoted in outputs. In this paper, we study T0, a large-scale multi-task text-to-text language model trained using prompt-based learning. We consider two different forms of semantically equivalent inputs: question-answer format and premise-hypothesis format. We use an existing bias benchmark for the former BBQ and create the first bias benchmark in natural language inference BBNLI with hand-written hypotheses while also converting each benchmark into the other form. The results on two benchmarks suggest that given two different formulations of essentially the same input, T0 conspicuously acts more biased in question answering form, which is seen during training, compared to premise-hypothesis form which is unlike its training examples. Code and data are released under https://github.com/feyzaakyurek/bbnli.

Motivation & Objective

• To investigate whether the form in which a problem is encoded influences social biases in large language models, independent of content.
• To compare bias levels in question-answering (QA) versus natural language inference (NLI) input formats for the same semantic inputs.
• To create and release BBNLI, the first bias benchmark for natural language inference with hand-written hypotheses, to support systematic evaluation of cognitive and social biases.
• To disentangle the effects of input form from content in multi-task learning, particularly in models like T0 trained on diverse prompts.
• To provide insights into mitigating bias through alternative input formulations without sacrificing model performance.

Proposed method

• The authors use T0, a 11B-parameter multi-task text-to-text model fine-tuned on diverse prompts across 62 datasets and 12 tasks.
• They evaluate bias using two benchmarks: BBQ (existing QA bias benchmark) and BBNLI (new NLI bias benchmark with hand-written hypotheses for 16 harmful stereotypes across three domains).
• Each example in BBNLI is provided in both QA and NLI forms, enabling direct comparison of model behavior across input formats while preserving semantic equivalence.
• The authors convert existing BBQ examples into NLI format (BBQ → NLI) and create a reverse benchmark (BBNLI → QA) to ensure balanced comparison across input forms.
• Bias is measured by computing the model’s preference for stereotypical over anti-stereotypical outputs across both formats.
• All code and datasets are released at https://github.com/feyzaakyurek/bbnli to support reproducibility and further research.

Experimental results

Research questions

• RQ1 Does the form of input (e.g., question-answer vs. premise-hypothesis) influence the level of social bias in a multi-task language model, independent of content?
• RQ2 How do bias levels compare between input formats that were present in training (e.g., QA) versus those that were not (e.g., NLI)?
• RQ3 To what extent does the model’s behavior shift when the same semantic input is reformulated into different prompt templates?
• RQ4 Can a new benchmark for natural language inference (BBNLI) effectively capture and measure cognitive and social biases in multi-task models?
• RQ5 Can alternative input formulations reduce bias without compromising model performance?

Key findings

• T0 exhibits significantly higher social bias in question-answering (QA) format compared to premise-hypothesis (NLI) format, despite identical semantic content.
• The bias difference is not due to performance degradation, as model accuracy remains stable across both input forms.
• The model shows stronger bias in QA format because it aligns with training data patterns, suggesting that training data distribution shapes bias susceptibility.
• The BBNLI benchmark successfully captures 16 harmful social stereotypes across three domains, including complex logical structures involving skip-logic.
• The study demonstrates that input form can be a lever for bias mitigation, as NLI format—unlike training data—leads to reduced bias in T0.
• The release of BBNLI and associated datasets enables future research into bias in multi-task and zero-shot generalization settings.