[Paper Review] See It from My Perspective: How Language Affects Cultural Bias in Image Understanding

Amith Ananthram, Elias Stengel-Eskin+2 more|arXiv (Cornell University)|Jun 17, 2024

Language, Metaphor, and CognitionPsychology3 citations

TL;DR

This paper diagnoses a Western cultural bias in state-of-the-art vision-language models (VLMs), showing they perform significantly better on Western than Eastern cultural images across both objective and subjective tasks. The bias stems from imbalanced language pre-training—using more English than non-English languages like Chinese—whereas a more balanced pre-training mix reduces bias even when prompting in English, and prompting in Chinese further mitigates it, especially when Chinese was well-represented during pre-training.

ABSTRACT

Vision-language models (VLMs) can respond to queries about images in many languages. However, beyond language, culture affects how we see things. For example, individuals from Western cultures focus more on the central figure in an image while individuals from East Asian cultures attend more to scene context. In this work, we characterize the Western bias of VLMs in image understanding and investigate the role that language plays in this disparity. We evaluate VLMs across subjective and objective visual tasks with culturally diverse images and annotations. We find that VLMs perform better on the Western split than on the East Asian split of each task. Through controlled experimentation, we trace one source of this bias in image understanding to the lack of diversity in language model construction. While inference in a language nearer to a culture can lead to reductions in bias, we show it is much more effective when that language was well-represented during text-only pre-training. Interestingly, this yields bias reductions even when prompting in English. Our work highlights the importance of richer representation of all languages in building equitable VLMs.

Motivation & Objective

• To investigate whether state-of-the-art VLMs exhibit cultural bias in image understanding, particularly favoring Western over Eastern visual perspectives.
• To determine how language distribution in text-only pre-training affects VLM bias in multimodal understanding.
• To evaluate whether prompting in a non-English language (e.g., Chinese) can reduce Western bias in VLMs.
• To compare the effectiveness of multilingual pre-training versus multilingual prompting in mitigating cultural bias in VLMs.
• To analyze whether bias reduction is more effective when the target language is well-represented during pre-training versus during inference.

Proposed method

• Evaluate multiple off-the-shelf VLMs (e.g., LLaVA variants) on culturally diverse image understanding tasks, splitting data into Western and Eastern subsets.
• Train LLaVA-based VLMs using Llama2 and Baichuan2 LLMs pre-trained on either English-only or English/Chinese mixed corpora (2T tokens), then fine-tune with CLIP vision encoders.
• Measure performance differences between Western and Eastern image splits across three tasks: object identification, question answering, and art emotion classification.
• Conduct controlled experiments by prompting in English or Chinese to assess the impact of language on bias reduction.
• Use logit lens probing to analyze hidden representations and assess whether models internalize culturally specific visual associations, especially in Chinese.
• Compare results from models pre-trained on monolingual vs. bilingual text corpora, and from multimodal fusion using monolingual vs. bilingual text corpora.

Figure 1 : Whose perspective do VLMs model? Despite being multilingual , state-of-the-art VLMs exhibit a bias toward the imagery and perspectives of Western culture. In contrast, a more balanced language mix during text-only pre-training produces VLMs that are both multilingual and multicultural .

Experimental results

Research questions

• RQ1 Does state-of-the-art VLM performance vary significantly between Western and Eastern cultural image subsets in both objective and subjective tasks?
• RQ2 How does the language distribution in text-only pre-training affect VLMs’ cultural bias in image understanding?
• RQ3 Can prompting in a non-English language like Chinese reduce Western bias in VLMs, and if so, under what conditions is this effect strongest?
• RQ4 Is a balanced multilingual pre-training distribution more effective than multilingual prompting alone in reducing cultural bias?
• RQ5 Does including Chinese in the multimodal fusion corpus during training reduce Western bias, and how does this compare to using only English?

Key findings

• VLMs exhibit significant Western bias across all evaluated tasks: object identification, question answering, and art emotion classification, with performance consistently higher on Western image subsets.
• Using a more balanced language mix (e.g., English and Chinese) during text-only pre-training reduces Western bias in VLMs, even when inference is performed in English.
• Prompting in Chinese reduces bias, but the effect is substantially stronger when Chinese was well-represented during pre-training, indicating pre-training language mix is more critical than inference language.
• On subjective tasks like art emotion classification, bias reduction is more pronounced, but objective tasks such as object identification also show measurable improvements with balanced pre-training.
• Baichuan2-based models show stronger cultural alignment with Chinese perspectives than Llama2-based models, as evidenced by hidden state probing that reveals Baichuan2 captures culturally specific visual associations.
• Multimodal fusion with bilingual corpora does not consistently reduce bias, highlighting that pre-training language distribution is more impactful than the language of the fusion data.

Figure 2 : Our approach. Step 1 : We measure the Western bias of off-the-shelf ( $\text{OTS}_{i}$ ) VLMs on culturally diverse image understanding tasks by comparing their performance on each task’s Western and Eastern splits. Step 2 : We train comparable multilingual VLMs (mLLaVA). We explore three