[Paper Review] A Systematic Evaluation of Large Language Models of Code

Frank F. Xu, Uri Alon+2 more|arXiv (Cornell University)|Feb 26, 2022

Software Engineering Research16 citations

TL;DR

This paper presents PolyCoder, a 2.7B-parameter open-source language model trained exclusively on 249GB of multilingual code across 12 programming languages. Despite Codex's superior performance in most languages, PolyCoder achieves lower perplexity than Codex in C, demonstrating that dedicated multilingual code pretraining can outperform proprietary models in specific languages.

ABSTRACT

Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex (Chen et al., 2021)) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing open-source models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area.

Motivation & Objective

• To systematically evaluate existing code language models—including proprietary models like Codex and open-source models like GPT-J, GPT-Neo, GPT-NeoX, and CodeParrot—across multiple programming languages.
• To identify the limitations of current open-source models, particularly the lack of large-scale, multilingual code-only models trained on diverse programming languages.
• To close this gap by training and releasing PolyCoder, a 2.7B-parameter model trained exclusively on a multilingual code corpus of 249GB across 12 languages.
• To compare model performance across intrinsic (perplexity) and extrinsic (HumanEval) benchmarks, analyzing the impact of model size, training data composition, and inference temperature.
• To provide a transparent, publicly available alternative to proprietary models like Codex, enabling broader research access and reproducibility in code LMs.

Proposed method

• Trained PolyCoder on a single machine using 249GB of code from 12 programming languages, selected from GitHub repositories, using the GPT-2 architecture.
• Constructed a new evaluation dataset of 12 programming languages to measure perplexity, ensuring unseen test sets to avoid data leakage.
• Evaluated models on HumanEval for code generation from natural language, measuring Pass@1, Pass@10, and Pass@100 with varying inference temperatures.
• Compared perplexity across models on the same held-out test set for each language, focusing on intrinsic language modeling quality.
• Analyzed the impact of model size (160M to 20B parameters), training data mix (natural language + code vs. code-only), and inference temperature on generation quality.
• Used temperature sampling during inference to assess trade-offs between diversity and accuracy, with ablation studies across model sizes.

Experimental results

Research questions

• RQ1 How does the performance of open-source code language models compare to the proprietary Codex model across multiple programming languages?
• RQ2 Can a large, open-source, multilingual code-only language model outperform existing open-source models and even proprietary models in specific languages?
• RQ3 What is the impact of training data composition—specifically, whether models are trained on mixed natural language and code or exclusively on code—on downstream performance?
• RQ4 How does model size and training duration affect performance on code generation and language modeling tasks?
• RQ5 How does inference temperature influence the quality of code generation, and what optimal settings exist for different evaluation metrics?

Key findings

• PolyCoder outperforms Codex in terms of perplexity on the C programming language, achieving a lower perplexity score than all other models tested, including Codex.
• In the HumanEval benchmark, higher temperature settings improve Pass@100 scores, while lower temperatures yield better Pass@1 results, indicating a trade-off between diversity and confidence.
• For languages like Go, Java, Rust, Scala, and TypeScript, increasing model size from 2.7B to 20B (GPT-Neo to GPT-NeoX) yields minimal performance gains, suggesting model capacity saturation.
• PolyCoder outperforms similarly sized GPT-Neo 2.7B in C, JavaScript, Rust, Scala, and TypeScript, indicating that code-only pretraining can be more effective than mixed pretraining for certain languages.
• Perplexity trends on the intrinsic evaluation correlate strongly with HumanEval performance on Python, suggesting that perplexity is a reliable, low-cost proxy for downstream code generation quality.
• Codex’s strong performance across languages may be overestimated due to potential data leakage, as its exact training set remains unknown and may include test data.