[Paper Review] CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Shuai Lu, Daya Guo+20 more|arXiv (Cornell University)|Feb 9, 2021

Software Engineering ResearchComputer Science98 references415 citations

TL;DR

CodeXGLUE presents a benchmark dataset with 10 tasks across 14 datasets for program understanding and generation, plus three baseline systems (CodeBERT, CodeGPT, and an Encoder-Decoder framework) to evaluate and compare models.

ABSTRACT

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

Motivation & Objective

• Establish a diversified benchmark to accelerate ML research in code understanding and generation.
• Provide a unified platform for evaluation and model comparison across multiple programming tasks and languages.
• Offer ready-to-use baselines to lower barriers for researchers entering code intelligence research.

Proposed method

• Assemble 14 datasets covering 10 tasks across multiple languages, including newly introduced cloze tests, line-level code completion, code translation, code search, and documentation translation.
• Provide three baseline model families: a BERT-style encoder (CodeBERT), a GPT-style decoder (CodeGPT), and an Encoder-Decoder framework for seq2seq tasks.
• Use pretraining and fine-tuning of these baselines on CodeXGLUE tasks to establish strong performance baselines.
• Describe data processing steps such as tokenization, vocabulary choices, and filtering to ensure data quality for each task.
• Propose a platform for standardized evaluation and comparison across tasks and languages.
• Ground experiments on established datasets like BigCloneBench, POJ-104, CodeSearchNet, CONCODE, and Microsoft Docs translations.

Experimental results

Research questions

• RQ1 Can a unified benchmark across diverse code understanding and generation tasks drive progress in ML for code?
• RQ2 What are the baseline performance levels of standard pretrained models on a broad suite of code-related tasks?
• RQ3 How do code-specific pretrained models (CodeBERT, CodeGPT) compare to traditional baselines on clone detection, defect detection, code completion, and related tasks?

Key findings

• CodeBERT achieves strong results on clone detection with an overall score of 90.4 on BigCloneBench/CodeBERT vs RoBERTa baselines.
• CodeBERT and CodeGPT baselines demonstrate competitive performance across multiple tasks, illustrating the benefit of pretrained, language-aware representations for code.
• Defect detection results show CodeBERT yielding 62.08% accuracy, indicating pretrained models provide notable gains over traditional baselines.
• CTL results (cloze tests) show CodeBERT outperforms RoBERTa across multiple languages and CT settings, highlighting its strong code understanding capabilities.
• The benchmark reveals that pretrained models improve code-related tasks, but there is room to further leverage code structure for even better performance.