[Paper Review] Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques

Joel Escudé Font, Marta R. Costa‐jussà|arXiv (Cornell University)|Jan 10, 2019

Natural Language Processing Techniques19 references44 citations

TL;DR

The paper debiases word embeddings used in a Transformer NMT system to reduce gender bias in English-Spanish translation, while maintaining or improving BLEU scores.

ABSTRACT

Neural machine translation has significantly pushed forward the quality of the field. However, there are remaining big issues with the output translations and one of them is fairness. Neural models are trained on large text corpora which contain biases and stereotypes. As a consequence, models inherit these social biases. Recent methods have shown results in reducing gender bias in other natural language processing tools such as word embeddings. We take advantage of the fact that word embeddings are used in neural machine translation to propose a method to equalize gender biases in neural machine translation using these representations. Specifically, we propose, experiment and analyze the integration of two debiasing techniques over GloVe embeddings in the Transformer translation architecture. We evaluate our proposed system on the WMT English-Spanish benchmark task, showing gains up to one BLEU point. As for the gender bias evaluation, we generate a test set of occupations and we show that our proposed system learns to equalize existing biases from the baseline system.

Motivation & Objective

• Assess gender bias in neural machine translation using a targeted Occupations test set.
• Evaluate debiasing strategies on word embeddings (GloVe variants) within an NMT framework.
• Measure translation quality impacts (BLEU) alongside gender-bias metrics.
• Compare encoder-only, decoder-only, and encoder+decoder deployment of pre-trained embeddings.

Proposed method

• Adopt Transformer-based NMT (OpenNMT implementation) as baseline.
• Use pre-trained word embeddings (GloVe variants) in encoder, decoder, or both.
• Apply debiasing techniques: Bolukbasi-style hard debiasing and GN-GloVe gender-neutral embeddings.
• Train models with 512-dim embeddings and standard OpenNMT hyperparameters.
• Evaluate translations with BLEU on newstest2013 and assess gender bias with an Occupations test set.

Experimental results

Research questions

• RQ1 Does incorporating debiased or gender-neutral word embeddings in NMT reduce gender bias in translations?
• RQ2 How do different placements of pre-trained embeddings (encoder, decoder, or both) affect BLEU scores and bias mitigation?
• RQ3 To what extent do debiasing techniques preserve translation quality while reducing gender stereotypes in outputs?

Key findings

• BLEU scores show little degradation and, in some cases, improvement when using pre-trained embeddings, especially GN-GloVe in both encoder and decoder.
• GN-GloVe embeddings in Enc./Dec. achieve the highest BLEU among tested configurations (30.74).
• Debiased embeddings substantially increase correct gender translation in the Occupations test, e.g., accuracy for female pronoun predictions improves by up to ~30 percentage points over baseline.
• Hard-debiased GloVe and GN-GloVe generally outperform non-debiased baselines on gender-neutral translation tasks.
• Using pre-trained embeddings generally yields BLEU gains and maintains translation quality for English-Spanish NT.
• The Occupations test demonstrates context-based gender resolution improves with debiasing, while some occupation translations remain challenging.