[Paper Review] Assessing Multilingual Fairness in Pre-trained Multimodal Representations

Jialu Wang, Yang Liu+1 more|arXiv (Cornell University)|Jun 12, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces multilingual individual and group fairness metrics to evaluate pre-trained multimodal models like CLIP across languages. It finds that while semantic similarity is preserved across languages (individual fairness), severe accuracy disparities exist—especially for low-resource languages—leading to biased predictions across race, gender, and age groups, revealing critical fairness gaps in multilingual multimodal AI.

ABSTRACT

Recently pre-trained multimodal models, such as CLIP, have shown exceptional capabilities towards connecting images and natural language. The textual representations in English can be desirably transferred to multilingualism and support downstream multimodal tasks for different languages. Nevertheless, the principle of multilingual fairness is rarely scrutinized: do multilingual multimodal models treat languages equally? Are their performances biased towards particular languages? To answer these questions, we view language as the fairness recipient and introduce two new fairness notions, multilingual individual fairness and multilingual group fairness, for pre-trained multimodal models. Multilingual individual fairness requires that text snippets expressing similar semantics in different languages connect similarly to images, while multilingual group fairness requires equalized predictive performance across languages. We characterize the extent to which pre-trained multilingual vision-and-language representations are individually fair across languages. However, extensive experiments demonstrate that multilingual representations do not satisfy group fairness: (1) there is a severe multilingual accuracy disparity issue; (2) the errors exhibit biases across languages conditioning the group of people in the images, including race, gender and age.

Motivation & Objective

• To investigate whether pre-trained multilingual multimodal models treat all languages equally in terms of fairness.
• To address the lack of systematic fairness evaluation in multilingual multimodal representation learning.
• To formalize fairness notions—individual and group fairness—for multilingual multimodal settings using visual grounding as an anchor.
• To empirically demonstrate that multilingual models exhibit significant accuracy disparities across languages and intersectional demographic groups.
• To highlight the impact of visual representation choice on group fairness metrics, urging caution in downstream applications.

Proposed method

• Define multilingual individual fairness as requiring similar semantic text snippets in different languages to connect equally to the same image.
• Define multilingual group fairness as requiring equal predictive performance across all languages in downstream tasks.
• Use CLIP as the base model to encode images and multilingual text into shared embedding space for similarity computation.
• Leverage the FairFace dataset with human-annotated attributes (race, gender, age) to evaluate group fairness across language and demographic dimensions.
• Construct neutral, descriptive text prompts with identical numeric age values across languages to isolate linguistic bias from semantic variation.
• Measure similarity scores via cosine similarity between image and text embeddings to assess model alignment and fairness.

Experimental results

Research questions

• RQ1 To what extent do pre-trained multilingual multimodal models satisfy multilingual individual fairness, where semantically equivalent text in different languages should yield similar image alignments?
• RQ2 Does multilingual group fairness hold, such that models perform equally across all languages in downstream vision-and-language tasks?
• RQ3 How do accuracy disparities vary across languages when evaluating with protected attributes like race, gender, and age?
• RQ4 To what extent does the choice of visual representation affect group fairness metrics in multilingual settings?
• RQ5 Are there significant cross-lingual disparities in age classification, even when numeric age values are identical across languages?

Key findings

• Multilingual individual fairness is largely satisfied, as semantically equivalent text in different languages shows consistent alignment with the same images.
• Severe multilingual accuracy disparities exist: for infant age classification, accuracy is 5.8% in English but 89.4% in German and 91.6% in Japanese.
• Significant cross-lingual gender gaps emerge in age classification, with disparities between male and female groups across languages.
• The model exhibits strong bias toward high-resource languages like German and Japanese, while underperforming on low-resource languages such as French for certain demographic categories.
• Group fairness metrics are sensitive to the choice of visual representations, indicating that model design choices influence fairness outcomes.
• Even with identical numeric age values in text prompts, accuracy varies significantly across languages, indicating linguistic and cultural encoding biases in the model.