[Paper Review] Multi-lingual Evaluation of Code Generation Models

Ben Athiwaratkun, Sanjay Krishna Gouda+23 more|arXiv (Cornell University)|Oct 26, 2022

Software Engineering Research28 citations

TL;DR

The paper introduces MBXP, Multilingual HumanEval, and MathQA-X benchmarks for execution-based code generation across 10+ languages, and analyzes multilingual versus monolingual model performance, few-shot prompting, zero-shot translation, robustness, and code insertion/summarization tasks while releasing the datasets and code.

ABSTRACT

We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated using a scalable conversion framework that transpiles prompts and test cases from the original Python datasets into the corresponding data in the target language. Using these benchmarks, we are able to assess the performance of code generation models in a multi-lingual fashion, and discovered generalization ability of language models on out-of-domain languages, advantages of multi-lingual models over mono-lingual, the ability of few-shot prompting to teach the model new languages, and zero-shot translation abilities even on mono-lingual settings. Furthermore, we use our code generation model to perform large-scale bootstrapping to obtain synthetic canonical solutions in several languages, which can be used for other code-related evaluations such as code insertion, robustness, or summarization tasks. Overall, our benchmarks represents a significant step towards a deeper understanding of language models' code generation abilities. We publicly release our code and datasets at https://github.com/amazon-research/mxeval.

Motivation & Objective

• Develop scalable multilingual execution-based benchmarks for code generation beyond Python (MBXP, Multilingual HumanEval, MathQA-X).
• Assess how multilingual models generalize to out-of-domain languages and benefit from few-shot prompting and translation.
• Enable analyses of robustness, code insertion, summarization, and other code-related tasks using synthetic canonical solutions.
• Provide public tooling and datasets to support broad evaluation of multilingual code generation models.

Proposed method

• Convert Python-based execution datasets into multiple target languages via a rule-based transpiler that preserves prompts and test cases.
• Train decoder-only transformer models across multilingual and monolingual settings using 125M, 672M, 2.7B, and 13B parameter sizes with 210B tokens per language (mono) or 210B per language with 630B total (multilingual).
• Evaluate using pass@k with unbiased estimates, generating complete function bodies and checking correctness against language-specific test cases.
• Use large-scale bootstrapping to synthesize canonical solutions in multiple languages to support additional evaluations (insertion, robustness, or summarization).
• Explore zero-shot translation by prepending reference solutions in a source language to target-language prompts and analyze translation-driven performance.

Experimental results

Research questions

• RQ1 How do multilingual code generation models perform on out-of-domain languages compared to monolingual models with equal resources?
• RQ2 Does training on multiple languages confer advantages over monolingual training when model size is large enough?
• RQ3 Can few-shot prompting and zero-shot translation improve code generation in previously unseen languages?
• RQ4 What is the impact of multilingual data spillover on cross-language code generation and translation?
• RQ5 How do multilingual models fare in robustness, code insertion, and code summarization tasks across languages?

Key findings

• Multilingual models often outperform the best monolingual models of equal resource at sufficiently large sizes.
• Models exhibit non-zero success on out-of-domain languages due to cross-language spillover in training data.
• Few-shot prompting improves out-of-domain execution accuracy and reduces non-assertion errors.
• Models show zero-shot translation abilities; translation can enable solving previously hard tasks, sometimes even for monolingual models.
• Multilingual models generally improve robustness and code summarization performance over monolingual ones, with notable gains in out-of-domain scenarios.
• In code insertion, right-context information significantly boosts Pass@1 across languages.