[论文解读] Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate

Marcelo Prates, Pedro H. C. Avelar+1 more|arXiv (Cornell University)|Sep 6, 2018

Text Readability and Simplification被引用 14

一句话总结

本研究通过将12种语言的性别中立句子（例如：'He/She is an engineer'）翻译成英语，调查了谷歌翻译中的性别偏见，发现尽管这些职业在现实中的性别分布已知，但翻译系统在代词使用上仍表现出强烈的男性默认倾向，尤其是在STEM领域。作者证明谷歌翻译系统在翻译中系统性地低估了女性的出现频率，表明这种偏见超出了人口统计数据本身，呼吁在自然语言处理系统中采用去偏技术。

ABSTRACT

Recently there has been a growing concern about machine bias, where trained statistical models grow to reflect controversial societal asymmetries, such as gender or racial bias. A significant number of AI tools have recently been suggested to be harmfully biased towards some minority, with reports of racist criminal behavior predictors, Iphone X failing to differentiate between two Asian people and Google photos' mistakenly classifying black people as gorillas. Although a systematic study of such biases can be difficult, we believe that automated translation tools can be exploited through gender neutral languages to yield a window into the phenomenon of gender bias in AI. In this paper, we start with a comprehensive list of job positions from the U.S. Bureau of Labor Statistics (BLS) and used it to build sentences in constructions like "He/She is an Engineer" in 12 different gender neutral languages such as Hungarian, Chinese, Yoruba, and several others. We translate these sentences into English using the Google Translate API, and collect statistics about the frequency of female, male and gender-neutral pronouns in the translated output. We show that GT exhibits a strong tendency towards male defaults, in particular for fields linked to unbalanced gender distribution such as STEM jobs. We ran these statistics against BLS' data for the frequency of female participation in each job position, showing that GT fails to reproduce a real-world distribution of female workers. We provide experimental evidence that even if one does not expect in principle a 50:50 pronominal gender distribution, GT yields male defaults much more frequently than what would be expected from demographic data alone. We are hopeful that this work will ignite a debate about the need to augment current statistical translation tools with debiasing techniques which can already be found in the scientific literature.

研究动机与目标

• 调查类似谷歌翻译的机器翻译系统在代词选择中是否表现出性别偏见。
• 检验此类偏见是否反映现实职业中的性别分布，或是否超过该分布。
• 评估语言类型学（例如，性别中立语言与有性别标记语言）对翻译偏见的影响。
• 评估通过自动翻译系统检测社会性别刻板印象的可行性。
• 倡导在统计机器翻译流程中整合去偏技术。

提出的方法

• 使用美国劳工统计局（BLS）提供的职业名称构建性别中立句子，格式为'He/She is a [job]'。
• 利用谷歌翻译API，将这些句子从12种性别中立语言（例如，匈牙利语、中文、约鲁巴语）翻译成英语。
• 收集并分析翻译输出中男性、女性及性别中立代词的频率。
• 将翻译中女性代词的频率与BLS提供的各职业实际女性参与率进行对比。
• 使用'quiet'（内向的）和'guilty'（内疚的）等形容词，测试性别化描述词如何影响翻译中的代词选择。
• 通过比较匈牙利语、中文和巴斯克语等语言的翻译输出，评估不同语言间的偏见差异。

实验结果

研究问题

• RQ1 当将性别中立句子翻译成英语时，谷歌翻译是否在代词选择上表现出男性默认？
• RQ2 翻译中女性代词的频率与各职业实际女性参与率的相关性有多大？
• RQ3 不同源语言（尤其是性别中立语言）如何影响翻译输出中的性别偏见程度？
• RQ4 性别化形容词（例如'quiet'、'guilty'）是否在翻译中引发对女性或男性代词的不成比例使用？
• RQ5 机器翻译输出能否作为检测训练数据中嵌入的社会性别刻板印象的代理工具？

主要发现

• 谷歌翻译在代词使用上表现出强烈的男性默认倾向，尤其在STEM职业头衔的翻译中，即使源语言为性别中立。
• 翻译中男性代词的频率显著高于基于现实世界中该职业女性参与率所预期的水平。
• 例如，在女性参与率较低的职业（如'software developer'）中，谷歌翻译仍产生了高于预期的男性代词比例。
• 形容词如'quiet'和'desirable'在翻译中与更高比例的女性代词相关联，而'guilty'和'cruel'则几乎完全与男性代词关联。
• 匈牙利语的代词分布比中文更均衡，表明语言结构会影响偏见的传播。
• 巴斯克语和约鲁巴语通常产生性别中立的翻译，但巴斯克语也出现了高比例的无法识别性别的代词，表明语言特定的去偏检测挑战。