[Paper Review] How to Write a Bias Statement: Recommendations for Submissions to the Workshop on Gender Bias in NLP

Christian Hardmeier, Marta R. Costa‐jussà+3 more|arXiv (Cornell University)|Apr 7, 2021

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper provides actionable guidelines for writing bias statements in NLP research, emphasizing explicit discussion of normative assumptions and types of harm—such as allocational and representational harms—linked to social impact. It advocates for critical reflection on definitions of bias and their real-world consequences, using concrete examples to illustrate how bias statements can improve transparency and accountability in gender bias research at NLP workshops.

ABSTRACT

At the Workshop on Gender Bias in NLP (GeBNLP), we'd like to encourage authors to give explicit consideration to the wider aspects of bias and its social implications. For the 2020 edition of the workshop, we therefore requested that all authors include an explicit bias statement in their work to clarify how their work relates to the social context in which NLP systems are used. The programme committee of the workshops included a number of reviewers with a background in the humanities and social sciences, in addition to NLP experts doing the bulk of the reviewing. Each paper was assigned one of those reviewers, and they were asked to pay specific attention to the provided bias statements in their reviews. This initiative was well received by the authors who submitted papers to the workshop, several of whom said they received useful suggestions and literature hints from the bias reviewers. We are therefore planning to keep this feature of the review process in future editions of the workshop.

Motivation & Objective

• To encourage NLP researchers to explicitly address the social and ethical implications of bias in their work.
• To provide a structured framework for authors to articulate how their research relates to real-world harms and societal values.
• To promote critical reflection on definitions of bias and their limitations in capturing systemic inequities.
• To support peer review by integrating social science and humanities reviewers who evaluate bias statements for normative and contextual soundness.
• To institutionalize bias statements as a standard practice in NLP workshops, particularly at GeBNLP, to improve research accountability.

Proposed method

• Propose a standardized format for bias statements in NLP submissions, especially for workshops like GeBNLP.
• Categorize types of harm into allocational and representational harms, drawing from Blodgett et al. (2020).
• Encourage authors to explicitly justify why certain behaviors are considered harmful, based on normative values and social impact.
• Advocate for clarity in defining bias, including limitations of the chosen definition and its potential exclusions.
• Integrate bias statement review into the peer review process by assigning reviewers with social science and humanities expertise.
• Use real-world examples to illustrate how bias statements can clarify the societal consequences of NLP systems.

Experimental results

Research questions

• RQ1 How can NLP researchers better articulate the social and ethical implications of bias in their work?
• RQ2 What types of harm—allocational or representational—are most relevant to gender bias in NLP systems?
• RQ3 How can normative assumptions underlying definitions of bias be made explicit and critically examined?
• RQ4 In what ways can bias statements improve the transparency and accountability of NLP research?
• RQ5 How can peer review processes be adapted to evaluate the quality and depth of bias statements?

Key findings

• Bias statements help make normative assumptions about harm and fairness explicit, enabling more transparent and accountable research.
• Allocational harms occur when systems unfairly allocate resources or opportunities across social groups, such as in gendered hiring tools.
• Representational harms arise when systems misrepresent, misgender, or erase certain social groups, such as non-binary or transgender individuals.
• Defining bias as harmful requires critical reflection on whether the definition captures all relevant injustices or overlooks systemic inequities.
• The inclusion of bias statements in peer review led to useful feedback and literature suggestions, indicating their value in improving research quality.
• The initiative was well-received by authors and is planned for continued use in future GeBNLP workshops.