[Paper Review] Mitigating Gender Bias in Machine Translation through Adversarial Learning

Eve Fleisig, Christiane Fellbaum|arXiv (Cornell University)|Mar 20, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper proposes an adversarial learning framework to mitigate gender bias in neural machine translation by training a gender adversary during fine-tuning of large language models. It achieves 86–91% reduction in gender bias disparity between male and female entity translations with minimal impact on overall translation quality, using two gender-probe methods: gender direction and pronoun usage.

ABSTRACT

Machine translation and other NLP systems often contain significant biases regarding sensitive attributes, such as gender or race, that worsen system performance and perpetuate harmful stereotypes. Recent preliminary research suggests that adversarial learning can be used as part of a model-agnostic bias mitigation method that requires no data modifications. However, adapting this strategy for machine translation and other modern NLP domains requires (1) restructuring training objectives in the context of fine-tuning pretrained large language models and (2) developing measures for gender or other protected variables for tasks in which these attributes must be deduced from the data itself. We present an adversarial learning framework that addresses these challenges to mitigate gender bias in seq2seq machine translation. Our framework improves the disparity in translation quality for sentences with male vs. female entities by 86% for English-German translation and 91% for English-French translation, with minimal effect on translation quality. The results suggest that adversarial learning is a promising technique for mitigating gender bias in machine translation.

Motivation & Objective

• To address gender bias in neural machine translation systems that reproduce harmful societal stereotypes.
• To develop a model-agnostic, data-free method for bias mitigation applicable to fine-tuned large language models.
• To define and measure gender as a protected variable in NLP tasks where gender is not pre-labeled.
• To evaluate the effectiveness of adversarial learning in reducing both representational and allocative gender bias in translation.
• To ensure bias mitigation does not degrade overall translation performance.

Proposed method

• Introduces two methods to infer gender as a protected variable: the gender direction method, which uses a gender subspace from word embeddings, and the pronoun usage method, which tracks gendered pronoun patterns.
• Trains an adversarial discriminator to predict gender from the model’s hidden representations during fine-tuning.
• Optimizes the main translation model to minimize the adversary’s ability to predict gender, thereby encouraging gender-invariant representations.
• Applies the framework during fine-tuning of T5-based sequence-to-sequence models for English-German and English-French translation.
• Uses the WinoMT dataset to evaluate gender bias on stereotypical and counter-stereotypical profession references.
• Employs F1 score disparity and accuracy metrics to quantify bias reduction and translation quality.

Experimental results

Research questions

• RQ1 How can gender be effectively defined and measured as a protected variable in large language models when not pre-labeled?
• RQ2 Can adversarial learning be successfully adapted to the pre-training/fine-tuning paradigm in modern NLP systems?
• RQ3 To what extent does adversarial training reduce gender bias in machine translation without degrading translation quality?
• RQ4 How do different gender-probe methods (gender direction vs. pronoun usage) compare in bias mitigation effectiveness?
• RQ5 Does bias mitigation via adversarial training improve generalization or translation robustness?

Key findings

• The gender direction method reduced gender bias disparity in translation quality by 86% for English-German and 91% for English-French translation.
• The pronoun usage method reduced the F1 score gap between male and female entity translations, with a 75% reduction in bias disparity on average.
• Translation accuracy improved slightly in most cases, suggesting potential generalization gains from bias mitigation.
• The model preserved the gender of entities in translation with higher accuracy after adversarial training.
• The method maintained high translation quality, with no significant degradation in BLEU or other standard metrics.
• Both gender-probe methods effectively reduced allocative and representational bias, with the gender direction method showing stronger but less consistent results.