[Paper Review] Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus

Luisa Bentivogli, Beatrice Savoldi+4 more|arXiv (Cornell University)|Jun 10, 2020

Natural Language Processing Techniques23 references4 citations

TL;DR

This paper introduces MuST-SHE, a multilingual benchmark for evaluating gender bias in speech translation (ST), comparing cascade and end-to-end ST systems on English-to-Italian and English-to-French directions. It finds that end-to-end models better leverage audio cues to reduce gender bias in speaker-self-referential contexts, while cascade models rely on external gender tags for similar gains, highlighting audio's potential to mitigate gender bias in ST.

ABSTRACT

Translating from languages without productive grammatical gender like English into gender-marked languages is a well-known difficulty for machines. This difficulty is also due to the fact that the training data on which models are built typically reflect the asymmetries of natural languages, gender bias included. Exclusively fed with textual data, machine translation is intrinsically constrained by the fact that the input sentence does not always contain clues about the gender identity of the referred human entities. But what happens with speech translation, where the input is an audio signal? Can audio provide additional information to reduce gender bias? We present the first thorough investigation of gender bias in speech translation, contributing with: i) the release of a benchmark useful for future studies, and ii) the comparison of different technologies (cascade and end-to-end) on two language directions (English-Italian/French).

Motivation & Objective

• To investigate whether speech translation systems can reduce gender bias by leveraging audio signals, which may contain speaker gender cues absent in text-only inputs.
• To address the lack of natural, balanced, and fine-grained benchmarks for evaluating gender bias in speech translation.
• To compare the performance of cascade (ASR + MT) and end-to-end ST systems on gendered language phenomena, particularly in speaker-self-referential contexts.
• To release MuST-SHE, a publicly available, multilingual benchmark with ~1,000 annotated audio-transcript-translation triplets for English-French and English-Italian.
• To develop a refined evaluation method that isolates gender-related performance from overall system accuracy, enabling more accurate bias measurement.

Proposed method

• The authors created MuST-SHE, a subset of the MuST-C corpus, containing ~1,000 (audio, transcript, translation) triplets annotated for gender-related phenomena across two language pairs: English-Italian and English-French.
• The corpus is structured into two categories: Category 1 (speaker refers to themselves) and Category 2 (speaker refers to others), enabling analysis of speaker-dependent vs. third-party gender cues.
• The study evaluates both cascade ST (ASR + MT) and end-to-end ST systems, with the latter trained directly on audio-text pairs to exploit acoustic features.
• A novel evaluation protocol was introduced, using the 'Diff' metric to measure gender bias by comparing performance on masculine vs. feminine gender classes.
• For cascade systems, an oracle variant (Cascade+Tag) was used, injecting speaker gender information via external tags to assess the impact of such knowledge.
• The evaluation focuses on gender-specific translation errors, particularly in pronoun and noun agreement, using BLEU scores adjusted to isolate gender-related performance.

Experimental results

Research questions

• RQ1 Can speech translation systems leverage audio signals to reduce gender bias in translation, especially when gender information is not explicitly present in the text?
• RQ2 How do end-to-end ST systems compare to cascade ST systems in handling speaker-self-referential gender phenomena, where audio may provide speaker identity cues?
• RQ3 To what extent does the injection of speaker gender information (via tags) improve cascade system performance on gendered translations?
• RQ4 How does the performance of ST systems vary across different gender phenomena, particularly in contexts where gender is inferred from discourse vs. audio cues?
• RQ5 Does the use of audio input in ST lead to a measurable reduction in gender bias compared to text-only MT, especially in languages with rich gender agreement systems?

Key findings

• End-to-end ST systems outperform cascade systems in Category 1 (speaker-self-reference), where audio cues can reveal speaker gender, achieving significantly lower gender bias (lower 'Diff' values) on the feminine class.
• Cascade systems exhibit strong gender bias in Category 1, particularly favoring masculine forms, with negative 'Diff' values on the feminine class indicating systematic mistranslation.
• In Category 2 (referring to others), where gender information is in the text, end-to-end systems perform worse than cascade systems, likely due to smaller training data and lack of robust audio features for gender recognition.
• The Cascade+Tag system, which injects speaker gender information, achieves the best performance in Category 1, demonstrating that external gender knowledge can compensate for the lack of audio-based inference.
• The MuST-SHE benchmark reveals that systems like End2End and Cascade+Tag are particularly challenged when speaker gender does not align with the target gender in translation, due to noisy or misleading audio-based gender signals.
• The study confirms that current ST systems remain affected by gender bias, but audio input offers a tangible advantage in speaker-self-referential contexts, suggesting audio can help mitigate bias when properly leveraged.