[Paper Review] Dynamic Neural Program Embedding for Program Repair

Ke Wang, Rishabh Singh+1 more|arXiv (Cornell University)|Nov 20, 2017

Software Engineering Research12 references54 citations

TL;DR

The paper proposes semantic dynamic program embeddings learned from runtime execution traces to improve program analysis tasks, including error classification and repair efficiency, outperforming syntax-based embeddings.

ABSTRACT

Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, fault localization, etc. However, most existing program embeddings are based on syntactic features of programs, such as raw token sequences or abstract syntax trees. Unlike images and text, a program has an unambiguous semantic meaning that can be difficult to capture by only considering its syntax (i.e. syntactically similar pro- grams can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited. This paper proposes a novel semantic program embedding that is learned from program execution traces. Our key insight is that program states expressed as sequential tuples of live variable values not only captures program semantics more precisely, but also offer a more natural fit for Recurrent Neural Networks to model. We evaluate different syntactic and semantic program embeddings on predicting the types of errors that students make in their submissions to an introductory programming class and two exercises on the CodeHunt education platform. Evaluation results show that our new semantic program embedding significantly outperforms the syntactic program embeddings based on token sequences and abstract syntax trees. In addition, we augment a search-based program repair system with the predictions obtained from our se- mantic embedding, and show that search efficiency is also significantly improved.

Motivation & Objective

• Demonstrate the fundamental limitations of syntax-based program representations for capturing semantics.
• Introduce dynamic program embeddings learned from program execution traces (variable trace, state trace, and dependency-enforced) to capture semantics.
• Evaluate embeddings on predicting common student error patterns in programming assignments.
• Show that dynamic embeddings improve search-based program repair efficiency when guided by semantic predictions.

Proposed method

• Represent program execution as dynamic traces: variable traces, state traces, and a dependency-enforced hybrid.
• Encode traces with GRU-based RNNs and combine via pooling to produce program embeddings.
• Enforce data/control dependencies across variable traces to better capture program semantics.
• Train and evaluate embeddings on predicting error patterns and integrating into SarfGen for guided repair.
• Compare dynamic embeddings to syntactic baselines: run-time trace, token, and AST encoders.

Experimental results

Research questions

• RQ1 Can dynamic embeddings learned from program execution traces outperform syntax-based embeddings for error-pattern prediction?
• RQ2 Do dynamic embeddings improve efficiency of a search-based program repair system when used to prioritize corrections?
• RQ3 Which embedding strategy (variable trace, state trace, or dependency-enforced) best captures program semantics for repair tasks?
• RQ4 How well do dynamic embeddings generalize across different programming problems and datasets?

Key findings

• Dynamic embeddings significantly outperform syntax-based embeddings in predicting common student error patterns (accuracy > 92% vs < 27%).
• Dependency-enforced embeddings yield strong semantics-aware representations by integrating variable dependencies during encoding.
• Using dynamic embeddings to guide SarfGen yields substantial speedups in repair time, especially as the number of fixes grows.
• Syntax-based traces (run-time) perform poorly relative to dynamic traces, highlighting semantic gaps in syntactic representations.