[Paper Review] MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks

Sanchit Ahuja, Divyanshu Aggarwal+9 more|arXiv (Cornell University)|Nov 13, 2023

Natural Language Processing Techniques4 citations

TL;DR

MEGAV ERS E introduces a comprehensive multilingual and multimodal benchmark comprising 22 datasets across 81 languages, including low-resource African languages, to evaluate state-of-the-art LLMs like GPT-4, PaLM2, and LLaVA-v1.5. It reveals that GPT-4 outperforms other models, especially on low-resource languages, while highlighting critical issues like dataset contamination and the need for better evaluation metrics beyond ROUGE-L.

ABSTRACT

There has been a surge in LLM evaluation research to understand LLM capabilities and limitations. However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored. Several new LLMs have been introduced recently, necessitating their evaluation on non-English languages. This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on the same set of multilingual datasets. Our benchmark comprises 22 datasets covering 83 languages, including low-resource African languages. We also include two multimodal datasets in the benchmark and compare the performance of LLaVA models, GPT-4-Vision and Gemini-Pro-Vision. Our experiments show that larger models such as GPT-4, Gemini-Pro and PaLM2 outperform smaller models on various tasks, notably on low-resource languages, with GPT-4 outperforming PaLM2 and Gemini-Pro on more datasets. We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs.

Motivation & Objective

• To expand multilingual LLM evaluation beyond English by including 6 new datasets, including low-resource African languages and two multimodal datasets.
• To benchmark state-of-the-art LLMs—GPT-3.5-Turbo, GPT-4, PaLM2, Llama2 (3 variants), and LLaVA-v1.5—across diverse tasks and languages.
• To address the lack of comprehensive evaluation for non-English and multimodal capabilities in existing benchmarks.
• To identify performance disparities, particularly for low-resource and non-Latin script languages, and to highlight issues like dataset contamination.
• To release code and data to support future research in multilingual and multimodal LLM evaluation.

Proposed method

• Extends the MEGA benchmark suite by integrating six new datasets, including two multimodal ones, to form MEGAV ERS E, covering 81 languages across 22 datasets.
• Evaluates five SOTA LLMs—GPT-4, PaLM2, Llama2 (three versions), GPT-3.5-Turbo, and LLaVA-v1.5—using standardized prompting strategies.
• Employs monolingual prompting for multilingual tasks and evaluates multimodal performance on image captioning and reasoning tasks.
• Uses standard metrics such as accuracy, ROUGE-L, and chrF++ for generation and classification tasks, while identifying limitations of ROUGE-L for summarization.
• Performs contamination analysis on MEGA datasets for GPT-4, though not for new datasets or models like PaLM2 and Llama2.
• Releases all code and evaluation scripts to enable reproducibility and further research in multilingual and multimodal LLM evaluation.

Figure 11: Results for XNLI across all languages and models for monolingual prompting

Experimental results

Research questions

• RQ1 How do leading LLMs such as GPT-4, PaLM2, and Llama2 perform across 81 languages, including low-resource African languages?
• RQ2 What is the comparative performance of multimodal models like LLaVA-v1.5 on multilingual multimodal tasks such as image captioning and visual reasoning?
• RQ3 To what extent do model performance disparities persist across languages with non-Latin scripts and low-resource settings?
• RQ4 How do standard evaluation metrics like ROUGE-L and chrF++ reflect actual generation quality, especially in low-resource settings?
• RQ5 What role does dataset contamination play in inflating performance scores, particularly in non-English language evaluations?

Key findings

• GPT-4 outperforms PaLM2 and Llama models across most multilingual tasks, particularly on low-resource languages and non-Latin script languages.
• PaLM2 outperforms GPT-4 on XNLI, PAWS-X, and Belebele, but GPT-4 leads on 7 out of 11 core multilingual tasks including TyDiQA and XQuaD.
• Llama2 models show particularly poor performance on Indian and African languages, indicating limited readiness for deployment without fine-tuning.
• ROUGE-L scores are insufficient for evaluating summarization quality, as many reasonable outputs receive low scores, especially in low-resource settings.
• The LLaVA-v1.5 model performs well on image captioning in high-resource languages but struggles on visual reasoning tasks, indicating a gap in multimodal reasoning across languages.
• Dataset contamination remains a critical issue, especially in non-English benchmarks, and future work must prioritize contamination detection and prevention.

Figure 12: Results for Indic-XNLI across all languages for monolingual prompting