[Paper Review] SCELMo: Source Code Embeddings from Language Models

Rafael Michael Karampatsis, Charles Sutton|arXiv (Cornell University)|Apr 28, 2020

Software Engineering ResearchComputer Science29 references34 citations

TL;DR

SCELMo trains ELMo-style contextual embeddings on JavaScript code and uses them to improve bug detection in DeepBugs, outperforming static embeddings and non-contextual baselines. It also analyzes OOV handling and generalization to external projects.

ABSTRACT

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engineering. We introduce a new set of deep contextualized word representations for computer programs based on language models. We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018). We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.

Motivation & Objective

• Motivate learning rich, contextual representations for source code to improve software engineering tasks.
• Apply the ELMo framework to source code to create context-aware token embeddings.
• Evaluate whether contextualized code embeddings improve bug detection performance over static embeddings.
• Assess generalization to unseen projects and the impact of out-of-vocabulary tokens.

Proposed method

• Train ELMo on a corpus of JavaScript code (150k files; filtered to 64,750 training and 33,229 validation files).
• Tokenize code with esprima and apply duplication filtering to reduce inflation.
• Use an embedding size of 100 features per forward/backward LSTM (200 per layer) and combine layer representations with equal weights (s_j = 1/(L+1), γ = 1).
• Integrate SCELMo representations into the DeepBugs bug detector as contextual features for three bug types.
• Compare against random, Word2Vec, FastText, and No-Context ELMo baselines to assess gains.
• Explore two querying variants: a fixed-context No-Context ELMo baseline and full SCELMo contextual embeddings for specific code constructs.

Experimental results

Research questions

• RQ1 Do SCELMo contextual embeddings improve bug-detection accuracy over static embeddings in DeepBugs?
• RQ2 How does SCELMo handle out-of-vocabulary tokens and complex/nested expressions compared to non-contextual approaches?
• RQ3 Does SCELMo generalize to unseen projects beyond the training corpus?
• RQ4 What is the impact of using contextual embeddings on real-bug mining scenarios and practical false-positive rates?

Key findings

• SCELMo substantially outperforms all static baselines on bug detection in both simple and complex expressions.
• On a validation set with single-name expressions, SCELMo achieves 92.11% vs. 86.18% random and 89.55% Word2Vec for Swapped Arguments, 100.00% vs. 90.47% random for Wrong Binary Operator, and 84.23% vs. 75.56% random for Wrong Binary Operand.
• Including complex expressions, SCELMo still dominates all baselines with 92.27% vs. 86.37% random for Swapped Arguments, 100.00% vs. 91.12% random for Wrong Binary Operator, and 87.10% vs. 72.73% random for Wrong Binary Operand.
• On an external test set of 500 JavaScript projects, SCELMo provides notable gains over static embeddings (e.g., 84.25% vs. 75.79% for Swapped Arguments, 99.99% vs. 82.95% for Wrong Binary Operator, 83.59% vs. 67.46% for Wrong Binary Operand).
• Non-contextual ELMo already surpasses static baselines, indicating value from pretraining and subword handling; SCELMo yields the largest improvements, especially with OOV tokens and complex expressions.
• OOV analysis shows substantial presence of OOV tokens in baseline components, highlighting SCELMo’s advantage in handling unseen identifiers and operands.