[Paper Review] Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code

Rafael-Michael Karampatsis, Hlib Babii+3 more|arXiv (Cornell University)|Mar 17, 2020

Software Engineering Research82 references76 citations

TL;DR

The paper analyzes vocabulary design for source-code language models, introduces an open-vocabulary NLM using BPE, scales to large corpora, and outperforms prior models on Java, C, and Python.

ABSTRACT

Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.

Motivation & Objective

• Study how vocabulary design choices affect vocabulary size and OOV rate in code language models.
• Develop and evaluate a large-scale open-vocabulary neural language model for source code using subword units.
• Demonstrate that open-vocabulary models outperform state-of-the-art models on multiple programming languages.
• Assess the impact of the open-vocabulary model on downstream tasks such as code completion and buggy code highlighting.

Proposed method

• Systematically evaluate vocabulary design choices (comments, strings, whitespace, filtering, and token splitting) for source code.
• Propose an open-vocabulary NLM that uses Byte-Pair Encoding to generate subword units.
• Train a GRU-based RNN LM on up to 13,362 projects across Java, C, and Python.
• Compare open-vocabulary NLM performance against n-gram LMs and closed-vocabulary NLMs on code completion.
• Evaluate the transfer of language-model improvements to downstream tasks like buggy code highlighting.
• Describe and release datasets, code, and trained models publicly.

Experimental results

Research questions

• RQ1 How do vocabulary design choices influence vocabulary size and OOV rate in source code language models?
• RQ2 Can an open-vocabulary model with subword units scale to large code corpora and diverse languages?
• RQ3 Do open-vocabulary models improve code completion performance across Java, C, and Python compared to baselines?
• RQ4 Do improvements in language modeling translate to improvements in downstream SE tasks such as bug detection/hunting?

Key findings

• Vocabulary design choices substantially affect vocabulary size and OOV rates, with large effects observed across experiments.
• Subword-based approaches beyond simple convention splitting are necessary to manage vocabulary effectively.
• Byte-Pair Encoding (BPE) enables a truly open vocabulary and reduces OOV issues while maintaining predictive power.
• Open-vocabulary NLMs outperform both n-gram LMs and closed-vocabulary NLMs for code completion across multiple languages.
• Open-vocabulary models also show superior performance in highlighting buggy code, indicating transfer to downstream SE tasks.
• The approach scales to training on thousands of projects, making it the largest reported neural models for code at the time.