[Paper Review] Structural Language Models of Code

Uri Alon, Roy Sadaka+2 more|arXiv (Cornell University)|Sep 30, 2019

Software Engineering Research44 citations

TL;DR

The paper introduces Structural Language Modeling (SLM) to perform any-code completion by predicting code as abstract syntax tree (AST) nodes via multiple AST paths, achieving state-of-the-art results on Java any-code completion and strong gains on C# restricted completion.

ABSTRACT

We address the problem of any-code completion - generating a missing piece of source code in a given program without any restriction on the vocabulary or structure. We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM). SLM estimates the probability of the program's abstract syntax tree (AST) by decomposing it into a product of conditional probabilities over its nodes. We present a neural model that computes these conditional probabilities by considering all AST paths leading to a target node. Unlike previous techniques that have severely restricted the kinds of expressions that can be generated in this task, our approach can generate arbitrary code in any programming language. Our model significantly outperforms both seq2seq and a variety of structured approaches in generating Java and C# code. Our code, data, and trained models are available at http://github.com/tech-srl/slm-code-generation/ . An online demo is available at http://AnyCodeGen.org .

Motivation & Objective

• Motivate the any-code completion problem where vocab and structure are unrestricted.
• Propose a structural language modeling approach that treats code as an AST and predicts node-by-node.
• Show that joint source-target modeling over AST paths improves generation quality over sequence- and other structured baselines.
• Demonstrate state-of-the-art results on Java any-code completion and strong gains on C# restricted completion.
• Provide analysis of ablations to identify components essential for performance.

Proposed method

• Represent a program as an AST and decompose Pr(A_P) into a product of conditional node probabilities across an AST traversal.
• Represent partial trees using sets of root-to-leaf paths and encode each path with an LSTM on node embeddings.
• Aggregate multiple path encodings with a transformer-based context and an index-informed root-path encoding to predict the next node.
• Predict the next AST node or subtokens using a syntactic copy mechanism that combines copy scores from path encodings with subtoken embeddings.
• Augment the generation with EOS nodes/tokens to control arity and depth in tree generation.
• Train end-to-end with cross-entropy using Adam, and perform beam search during inference; compare against NMT and code-structure baselines.

Experimental results

Research questions

• RQ1 Can any-code completion be effectively modeled by AST-path conditioned probabilities rather than flat sequences?
• RQ2 Does jointly modeling source and target code as the same tree improve generation quality over encoder-decoder or production-rule based approaches?
• RQ3 What is the impact of path-based representations, attention aggregation, and code-copy mechanisms on exact-match and tree-structure accuracy?
• RQ4 How does SLM perform on Java any-code completion and C# restricted completion relative to strong baselines?
• RQ5 What ablations reveal the contribution of components like root attention, copying, and path-based representations?

Key findings

• SLM achieves state-of-the-art exact-match acc@1 and acc@5 on Java any-code completion: 18.04% and 24.83% respectively, with tree@1 39.10% and tree@5 55.32%.
• On Java, SLM outperforms all baselines including code2seq, seq2tree, and Transformer variants, with notable gains in acc@1 and acc@5.
• In restricted C# completion, SLM reaches 37.61% acc@1 and 45.51% acc@5, and tree@1 51.10% with tree@5 59.82%, surpassing GNN→NAG and other baselines.
• Ablations show joint modeling (Paths→Paths) and copy mechanisms are crucial; removing root attention or copy reduces performance significantly (e.g., No Copy severely drops metrics).
• The tree@k metric indicates models often predict correct syntax even when subtokens differ, highlighting the potential for further gains through better token/name prediction.