[Paper Review] SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning

Bin Wang, Zhengyuan Liu+5 more|arXiv (Cornell University)|Sep 9, 2023

Topic Modeling4 citations

TL;DR

SeaEval introduces a comprehensive benchmark for multilingual foundation models, evaluating performance across classic NLP tasks, reasoning, cultural understanding, and cross-lingual consistency. Key findings reveal significant inconsistencies in model responses to semantically equivalent multilingual queries, exposure bias in label arrangements, and a lack of balanced multilingual proficiency despite training, underscoring the need for improved semantic generalization and multilingual contextualization.

ABSTRACT

We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we investigate the brittleness of foundation models in the dimensions of semantics and multilinguality. Our analyses span both open-sourced and closed models, leading to empirical results across classic NLP tasks, reasoning, and cultural comprehension. Key findings indicate (1) Most models exhibit varied behavior when given paraphrased instructions. (2) Many models still suffer from exposure bias (e.g., positional bias, majority label bias). (3) For questions rooted in factual, scientific, and commonsense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, most models surprisingly demonstrate inconsistent performance on these queries. (4) Multilingually-trained models have not attained "balanced multilingual" capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for more thorough investigations and evaluations for multilingual and multicultural scenarios.

Motivation & Objective

• To develop a holistic benchmark for evaluating multilingual foundation models beyond standard NLP tasks.
• To investigate model behavior across cultural reasoning and multilingual consistency, especially under semantically equivalent but linguistically diverse inputs.
• To identify and analyze brittleness in model responses due to semantic and multilingual inconsistencies.
• To assess whether multilingually-trained models achieve balanced proficiency across languages.
• To provide new datasets and metrics that fill gaps in cultural and cross-lingual evaluation.

Proposed method

• The benchmark includes 28 datasets, with 6 newly constructed for cultural reasoning and cross-lingual consistency.
• It evaluates models across four dimensions: classic NLP, complex reasoning, cultural understanding, and cross-lingual knowledge transfer.
• For cross-lingual consistency, the same factual or commonsense question is translated into multiple languages (e.g., English, Chinese, Indonesian, Spanish, etc.) to test response uniformity.
• Tailored metrics are introduced to assess semantic and multilingual robustness, including response consistency across paraphrased and multilingual inputs.
• The evaluation spans both open-source and closed-source foundation models across diverse linguistic and cultural contexts.
• A leaderboard and open-source toolkit are provided via GitHub to support reproducibility and ongoing evaluation.

Experimental results

Research questions

• RQ1 How consistently do multilingual foundation models respond to semantically equivalent but linguistically varied instructions across languages?
• RQ2 To what extent do models exhibit exposure bias, such as positional or majority label bias, in multilingual settings?
• RQ3 How well do models transfer knowledge across languages for factual, scientific, and commonsense questions when the meaning is preserved?
• RQ4 Do multilingually-trained models achieve balanced proficiency across all languages, or do they favor certain languages?
• RQ5 How effectively do models reason about cultural practices, values, and nuances embedded in language?

Key findings

• Many models produce inconsistent responses when given paraphrased instructions, indicating brittleness in semantic understanding.
• Exposure bias, including positional and majority label bias, remains prevalent across multiple models despite training.
• Models show inconsistent performance on semantically equivalent multilingual queries for factual and commonsense knowledge, suggesting non-generalized semantic representations.
• Multilingually-trained models fail to achieve 'balanced multilingual' capabilities, with performance varying significantly across languages.
• Cultural reasoning tasks reveal that models often misinterpret culturally specific cues, such as ritual behaviors or social signals in language.
• The benchmark reveals that current models lack robust cross-lingual consistency, even for simple factual questions, undermining trust in multilingual deployment.