[Paper Review] A Literature Study of Embeddings on Source Code

Zimin Chen, Martin Monperrus|arXiv (Cornell University)|Apr 5, 2019

Software Engineering ResearchComputer Science26 references56 citations

TL;DR

This survey collects and categorizes how word embeddings are applied to source code across multiple granularities and discusses publicly available data and visualization techniques.

ABSTRACT

Natural language processing has improved tremendously after the success of word embedding techniques such as word2vec. Recently, the same idea has been applied on source code with encouraging results. In this survey, we aim to collect and discuss the usage of word embedding techniques on programs and source code. The articles in this survey have been collected by asking authors of related work and with an extensive search on Google Scholar. Each article is categorized into five categories: 1. embedding of tokens 2. embedding of functions or methods 3. embedding of sequences or sets of method calls 4. embedding of binary code 5. other embeddings. We also provide links to experimental data and show some remarkable visualization of code embeddings. In summary, word embedding has been successfully applied on different granularities of source code. With access to countless open-source repositories, we see a great potential of applying other data-driven natural language processing techniques on source code in the future.

Motivation & Objective

• Survey the use of word embeddings on source code across multiple granularities (tokens, functions, API sequences, binaries, etc.).
• Categorize existing work into five embedding categories and summarize available experimental data.
• Highlight visualization approaches and public data resources to facilitate replication and further research.

Proposed method

• Review articles by contacting related authors and performing extensive Google Scholar searches.
• Categorize each work into five embedding categories: token, function/method, sequences/API usage, binary code, and other embeddings.
• Provide links to publicly available experimental data and visual representations of code embeddings.
• Summarize the potential and limitations of embeddings in code and discuss visualization techniques.

Experimental results

Research questions

• RQ1 What are the prevailing granularities at which code embeddings are applied (tokens, functions, API sequences, binaries, others)?
• RQ2 How have embedding techniques been used to support downstream code tasks (e.g., repair, clone detection, vulnerability prediction)?
• RQ3 What public data and visualizations exist to support replication and further study of code embeddings?
• RQ4 What future directions are suggested for contextual or downstream-task evaluated embeddings in source code?

Key findings

• Embedding techniques are applied to various granularities of code, including tokens, functions/methods, API sequences, and binary code.
• Many works use word2vec or similar methods to learn token, function, and API embeddings for downstream tasks like repair, clone detection, and vulnerability prediction.
• Several studies employ structured representations (e.g., ASTs, control-flow graphs, symbol traces) and neural networks to learn embeddings for code semantics.
• Visualization methods (PCA, t-SNE) are used to illustrate high-dimensional code embeddings.
• A number of public datasets and artifacts are linked to code embedding research to enable replication and reuse.