[Paper Review] Structured Generative Models of Natural Source Code

Chris J. Maddison, Daniel Tarlow|arXiv (Cornell University)|Jan 2, 2014

Software Engineering ResearchComputer Science28 references93 citations

TL;DR

This paper proposes structured generative models for natural source code (NSC) by extending probabilistic context-free grammars (PCFGs) and neuro-probabilistic language models with compiler-inspired reasoning and hierarchical structure modeling. The resulting Log-bilinear Tree-Traversal (LTT) models significantly outperform baselines in predictive log-likelihood on held-out code, generating more realistic and semantically coherent code samples by capturing scoping rules, block structures, and variable usage patterns.

ABSTRACT

We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs.

Motivation & Objective

• To develop generative models tailored to natural source code (NSC), which is human-written, hierarchical, and semantically structured.
• To improve code completion, bug detection, and code summarization by learning a unified probabilistic model of source code.
• To incorporate compiler-like reasoning into neural and probabilistic models to better capture language-specific constraints and scoping rules.
• To demonstrate that structured generative modeling leads to better performance and more realistic code samples than standard NLP or n-gram baselines.
• To explore the potential of generative models as a foundation for downstream tasks like program induction, API suggestion, and code representation learning.

Proposed method

• Extends probabilistic context-free grammars (PCFGs) with log-bilinear parameterizations to model hierarchical tree structures in source code.
• Introduces Log-bilinear Tree-Traversal (LTT) models that combine tree-structured generation with neural parameterization for efficient inference and training.
• Incorporates compiler-like reasoning by modeling local and global variable scopes, ensuring syntactically and semantically valid code generation.
• Uses a hierarchical sequence model (Hi&Seq) to model block statements and method bodies, improving structure modeling over flat sequence baselines.
• Employs a scope-aware model that distinguishes between local and global identifier tokens, reducing generation cost and improving accuracy.
• Trains models efficiently using a corpus of C# source code from public repositories, leveraging the Roslyn compiler’s rich AST and symbol information.

Experimental results

Research questions

• RQ1 Can structured generative models that combine PCFGs with neural parameterization outperform standard NLP and n-gram models in modeling natural source code?
• RQ2 How does incorporating compiler-like reasoning—such as scope tracking and type constraints—affect the quality of generated code?
• RQ3 To what extent do hierarchical and structural modeling components (e.g., for loops, blocks, method declarations) contribute to improved log-likelihood and realism?
• RQ4 Can the model learn subtle syntactic patterns, such as variable reuse and scoping rules, without explicit supervision?
• RQ5 How do different architectural components (e.g., scope modeling, block modeling) contribute to the overall predictive performance?

Key findings

• The LTT-HiSeq-Scope model achieves the best test log-likelihood of -2.33, significantly outperforming the baseline LBL n-gram model at -4.23.
• The model pays a higher cost for generating tree structure than the Hi&Seq baseline but compensates with a much lower cost for generating tokens, indicating better structural efficiency.
• IdentifierTokens from the global scope contribute the most to the total log probability (30.1%), followed by local scope identifiers (10.9%) and Block statements (10.6%), highlighting key modeling challenges.
• Qualitative samples show the model generates realistic for loops with proper scoping, variable reuse, and syntactic patterns like 'int' variables appearing in square brackets.
• The model captures high-level code structure, such as method declarations and block nesting, and learns subtle patterns like the use of 'Math.Max' in nested loops.
• The scope model reduces token generation cost by 1.68 compared to the baseline, demonstrating the benefit of structured modeling over flat sequence generation.