[Paper Review] Latent Predictor Networks for Code Generation

Ling Wang, Edward Grefenstette+5 more|arXiv (Cornell University)|Mar 22, 2016

Natural Language Processing Techniques28 references103 citations

TL;DR

Introduces Latent Predictor Networks (LPNs) to generate code conditioned on multiple input predictors, with structured attention and code compression, achieving state-of-the-art results on new TCG card datasets and a Django corpus.

ABSTRACT

Many language generation tasks require the production of text conditioned on both structured and unstructured inputs. We present a novel neural network architecture which generates an output sequence conditioned on an arbitrary number of input functions. Crucially, our approach allows both the choice of conditioning context and the granularity of generation, for example characters or tokens, to be marginalised, thus permitting scalable and effective training. Using this framework, we address the problem of generating programming code from a mixed natural language and structured specification. We create two new data sets for this paradigm derived from the collectible trading card games Magic the Gathering and Hearthstone. On these, and a third preexisting corpus, we demonstrate that marginalising multiple predictors allows our model to outperform strong benchmarks.

Motivation & Objective

• Motivate the need for conditioning output on multiple predictors beyond a single softmax.
• Propose Latent Predictor Networks (LPNs) to marginalize over predictor choices during training.
• Extend attention to structured inputs with multiple fields.
• Demonstrate code generation from natural language and structured specifications using new MTG/HS datasets.
• Show improvements over strong baselines and analyze components such as compression and pointer networks.

Proposed method

• Extend attention-based seq2seq models to attend over multiple input fields with varying sizes via a common projection and structured attention (Equations 4–5).
• Define Latent Predictor Networks to model marginal likelihood over a sequence of predictors and generated segments (Equation 7).
• Introduce three predictor types: character generation, copy-from-singular-field, and copy-from-text-field (pointer networks).
• Train with forward-backward dynamic programming to marginalize over latent predictor sequences (forward–backward for Semi-Markov models).
• Decode with a stack-based beam-search, merging paths that produce identical outputs.

Experimental results

Research questions

• RQ1 Can a neural architecture marginalize over multiple predictors to generate output, while handling segments of varying granularity?
• RQ2 Does structured attention over multiple input fields improve code generation from mixed natural language and structured specs?
• RQ3 Do LPNs with pointer networks outperform strong baselines on new MTG/HS datasets and Django corpus?
• RQ4 What is the impact of code compression on training efficiency and generation quality?
• RQ5 How accurately can the system copy essential card attributes into generated code?

Key findings

• LPN-based model plus structured attention and code compression outperforms baselines on MTG, HS, and Django datasets.
• On MTG, HS, and Django, Our System achieves higher BLEU and non-zero accuracy where baselines fail to produce valid code.
• Compression enables training on long-code sequences with negligible loss in quality, while preserving decoding performance.
• Pointer networks are crucial; removing LPN components degrades performance significantly.
• Combining character-level generation with copy mechanisms yields strong performance, especially for copying card names and numeric attributes.