[Paper Review] HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization

Qiwei Peng, Yekun Chai+1 more|arXiv (Cornell University)|Feb 26, 2024

Natural Language Processing Techniques4 citations

TL;DR

HumanEval-XL introduces a large-scale, multilingual code generation benchmark connecting 23 natural languages (NLs) and 12 programming languages (PLs), comprising 22,080 prompts with an average of 8.33 test cases per task. It enables comprehensive evaluation of cross-lingual natural language generalization in code generation, revealing that model size and specialized code pre-training significantly improve multilingual code generation, while current models struggle to preserve semantic equivalence across diverse NLs.

ABSTRACT

Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \url{https://github.com/FloatAI/humaneval-xl}.

Motivation & Objective

• To address the lack of comprehensive benchmarks for evaluating multilingual code generation across diverse natural languages and programming languages.
• To enable rigorous assessment of cross-lingual natural language generalization in large language models (LLMs) beyond English-centric or limited-language settings.
• To establish a standardized, parallel benchmark connecting 23 NLs and 12 PLs for evaluating multilingual LLMs' understanding and code generation capabilities.
• To investigate the impact of model scale and specialized code pre-training on multilingual code generation performance.

Proposed method

• The benchmark is constructed by creating parallel coding problems across 23 natural languages and 12 programming languages, ensuring linguistic and syntactic alignment.
• Each prompt is translated into 23 NLs and mapped to equivalent code in 12 PLs, forming a parallel multilingual dataset.
• The dataset includes 22,080 prompts with an average of 8.33 test cases per task, ensuring robust evaluation of code correctness.
• Evaluation uses the pass@k metric to measure the success rate of generated code on test cases.
• Experiments are conducted on three LLM families—CodeT5+, CodeGen2, GPT-3.5, and GPT-4—across varying parameter scales.
• The benchmark is publicly released at https://github.com/FloatAI/HumanEval-XL for reproducibility and community use.

Experimental results

Research questions

• RQ1 How well do large language models generalize across 23 diverse natural languages in code generation tasks?
• RQ2 What is the impact of model size on multilingual code generation performance across different programming languages?
• RQ3 How does specialized code pre-training affect cross-lingual code generation capabilities compared to general multilingual pre-training?
• RQ4 To what extent do current LLMs preserve semantic equivalence when translating the same coding task across different natural languages?
• RQ5 Can the benchmark detect linguistic biases or performance drops in low-resource natural languages during code generation?

Key findings

• Model size has a strong positive correlation with multilingual code generation performance, with larger models achieving significantly higher pass@k scores across all languages and programming languages.
• Specialized code pre-training plays a pivotal role in improving code generation quality, especially in low-resource language settings.
• Despite high performance in English, current LLMs struggle to maintain semantic equivalence when generating code from non-English prompts, indicating poor cross-lingual generalization.
• GPT-4 achieves the highest pass@1 scores across all languages and PLs, outperforming CodeT5+ and CodeGen2 models, particularly in low-resource language pairs.
• Many low-resource languages (e.g., Tagalog, Swahili, Uzbek) show near-zero performance across all models, highlighting a critical gap in multilingual model generalization.
• The benchmark reveals that even high-parameter models fail to generalize effectively across all 23 natural languages, underscoring the need for improved cross-lingual alignment in code LLMs.