[Paper Review] CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Zhangyin Feng, Daya Guo+9 more|arXiv (Cornell University)|Feb 19, 2020

Topic Modeling27 references247 citations

TL;DR

CodeBERT is a bimodal pre-trained Transformer model trained on natural language and code data, achieving state-of-the-art results on NL-code search and code documentation generation, and enabling zero-shot NL-PL probing.

ABSTRACT

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.

Motivation & Objective

• Develop a unified pre-trained model that spans natural language and programming languages across multiple programming languages.
• Leverage both bimodal NL-PL pairs and unimodal code data to learn robust representations.
• Demonstrate effectiveness on NL-PL tasks such as natural language code search and code documentation generation.
• Investigate what knowledge CodeBERT captures via NL-PL probing in a zero-shot setting.

Proposed method

• Use a Transformer-based architecture (RoBERTa-base size, 125M parameters).
• Pre-train with a hybrid objective combining Masked Language Modeling (MLM) on bimodal NL-PL data and Replaced Token Detection (RTD) that also leverages unimodal data.
• Represent inputs as two segments (NL and code) separated by a [SEP] token and use [CLS] for aggregate representations.
• Train on six languages (Python, Java, JavaScript, PHP, Ruby, Go) with bimodal NL-PL pairs and unimodal code data.
• Generate plausible alternative tokens using generators for RTD, and train a discriminator to distinguish original vs. replaced tokens.
• Fine-tune CodeBERT for downstream NL-PL tasks such as NL-based code search and code-to-text generation.

Experimental results

Research questions

• RQ1 Does a bimodal pre-trained model trained on NL-PL pairs and unimodal code data improve NL-PL understanding tasks over purely NL or code-only models?
• RQ2 What is the impact of MLM alone, RTD alone, and their combination on NL-PL tasks?
• RQ3 Can CodeBERT generalize to code search and code documentation generation across multiple programming languages?
• RQ4 How does CodeBERT perform in NL-PL probing compared to RoBERTa and code-only pre-trained models?

Key findings

• CodeBERT achieves state-of-the-art results on natural language code search (CodeSearchNet) when fine-tuned, outperforming RoBERTa and code-only pre-trained models.
• CodeBERT with MLM+RTD pre-training (initialized from RoBERTa) yields the best retrieval performance across languages (e.g., overall Ma-Avg improvements over baselines).
• In code documentation generation, CodeBERT-based encoders achieve higher BLEU-4 scores than RoBERTa and code-only baselines, with RTD+MLM providing further gains.
• NL-PL probing in a zero-shot setting shows CodeBERT outperforms RoBERTa and code-only pre-trained models in both PL and NL prediction tasks.
• CodeBERT demonstrates better generalization to a programming language not seen in pre-training (C# code-to-NL generation) than RoBERTa and some baselines, though not exceeding state-of-the-art code2seq in all cases.
• A case study on NL and PL probing illustrates that CodeBERT can correctly predict masked NL and PL tokens where RoBERTa fails.