[Paper Review] Maybe Deep Neural Networks are the Best Choice for Modeling Source Code

Rafael-Michael Karampatsis, Charles Sutton|arXiv (Cornell University)|Mar 13, 2019

Software Engineering Research63 references44 citations

TL;DR

The paper presents an open-vocabulary neural language model for source code using subword units learned by Byte Pair Encoding (BPE), achieving state-of-the-art results across Java, C, and Python and enabling fast adaptation to new projects.

ABSTRACT

Statistical language modeling techniques have successfully been applied to source code, yielding a variety of new software development tools, such as tools for code suggestion and improving readability. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. But traditional language models limit the vocabulary to a fixed set of common words. For code, this strong assumption has been shown to have a significant negative effect on predictive performance. But the open vocabulary version of the neural network language models for code have not been introduced in the literature. We present a new open-vocabulary neural language model for code that is not limited to a fixed vocabulary of identifier names. We employ a segmentation into subword units, subsequences of tokens chosen based on a compression criterion, following previous work in machine translation. Our network achieves best in class performance, outperforming even the state-of-the-art methods of Hellendoorn and Devanbu that are designed specifically to model code. Furthermore, we present a simple method for dynamically adapting the model to a new test project, resulting in increased performance. We showcase our methodology on code corpora in three different languages of over a billion tokens each, hundreds of times larger than in previous work. To our knowledge, this is the largest neural language model for code that has been reported.

Motivation & Objective

• Address the out-of-vocabulary problem for code identifiers in language modeling.
• Propose an open-vocabulary neural language model based on subword units to improve code modeling.
• Demonstrate state-of-the-art predictive performance across multiple languages and large-scale datasets.
• Show practical utility via a simple method for adapting the model to a new project.

Proposed method

• Use a single-layer GRU neural language model over subword units learned by BPE.
• Segment code tokens into subword units to create an open vocabulary.
• Train on large code corpora and evaluate with cross-entropy and MRR metrics.
• Provide a beam-search–like procedure to predict top-k complete tokens from subword sequences.
• Develop a simple dynamic adaptation procedure that updates the global model with a single gradient step on a new project.

Experimental results

Research questions

• RQ1 Can open-vocabulary neural language models (NLMs) outperform fixed-vocabulary models for code across multiple programming languages?
• RQ2 Do subword units learned via BPE improve handling of OOV identifiers in code?
• RQ3 Is it feasible to adapt a large, pre-trained code LM to a new project efficiently while preserving generalization?
• RQ4 How does the proposed model scale with corpus size and different languages (Java, C, Python)?

Key findings

• The open-vocabulary NLM with subword units outperforms prior neural and n-gram LMs for code across Java, C, and Python.
• On Java, the model achieves 3.15 bits per token and 70.84% MRR in cross-project evaluation and 1.04 bits per token and 81.16% MRR in within-project evaluation.
• The study uses corpora of over a billion tokens per language, making it the largest neural LM for code reported to date (1.7 billion tokens).
• The subword approach enables prediction of OOV identifiers by leveraging subtoken statistics that occur within training data.
• A simple adaptation method updates the global model with a single gradient step per encountered sequence to tailor it to a new project.