[Paper Review] Dynamic Neural Program Embeddings for Program Repair

Ke Wang, Zhendong Su+1 more|arXiv (Cornell University)|Feb 15, 2018

Software Engineering Research35 citations

TL;DR

This paper proposes a dynamic neural program embedding that captures program semantics through execution traces, modeling live variable states as sequential tuples for improved performance in program repair. Evaluated on student programming assignments and CodeHunt exercises, the semantic embedding significantly outperforms syntax-based embeddings and enhances search-based repair efficiency.

ABSTRACT

Neural program embeddings have shown much promise recently for a variety of program analysis tasks, including program synthesis, program repair, fault localization, etc. However, most existing program embeddings are based on syntactic features of programs, such as raw token sequences or abstract syntax trees. Unlike images and text, a program has an unambiguous semantic meaning that can be difficult to capture by only considering its syntax(i.e. syntactically similar programs can exhibit vastly different run-time behavior), which makes syntax-based program embeddings fundamentally limited. This paper proposes a novel semantic program embedding that is learned from program execution traces. Our key insight is that program states expressed as sequential tuples of live variable values not only captures program semantics more precisely, but also offer a more natural fit for Recurrent Neural Networks to model. We evaluate different syntactic and semantic program embeddings on predicting the types of errors that students make in their submissions to an introductory programming class and two exercises on the CodeHunt education platform. Evaluation results show that our new semantic program embedding significantly outperforms the syntactic program embeddings based on token sequences and abstract syntax trees. In addition, we augment a search-based program repair system with the predictions obtained from our semantic embedding, and show that search efficiency is also significantly improved.

Motivation & Objective

• To address the limitation of syntax-based program embeddings in capturing semantic differences between programs with similar structures.
• To explore whether execution traces—specifically sequences of live variable values—can serve as a more effective basis for learning program embeddings.
• To improve program repair by leveraging semantic embeddings to guide search-based repair systems more efficiently.
• To evaluate the effectiveness of semantic embeddings in real-world educational programming contexts involving student-submitted code.

Proposed method

• Learning program embeddings by modeling program execution traces as sequential tuples of live variable values during runtime.
• Using Recurrent Neural Networks (RNNs) to process and encode these sequential program states into dense semantic embeddings.
• Training the RNN-based embedding model on execution traces from program runs to capture dynamic semantic behavior.
• Comparing the performance of the proposed semantic embedding against syntax-based embeddings derived from raw token sequences and abstract syntax trees.
• Integrating the semantic embedding predictions into a search-based program repair system to guide the search space and improve efficiency.
• Evaluating the embedding quality via error type prediction tasks on real student programming submissions and CodeHunt exercises.

Experimental results

Research questions

• RQ1 Can program embeddings based on execution traces capture program semantics more effectively than syntax-based embeddings?
• RQ2 How does the proposed semantic embedding perform in predicting common error types in student programming assignments?
• RQ3 To what extent does the semantic embedding improve the efficiency of search-based program repair systems?
• RQ4 How do the performance gains of the semantic embedding compare across different types of programming tasks and error patterns?

Key findings

• The proposed semantic program embedding significantly outperforms syntax-based embeddings (based on token sequences and abstract syntax trees) in predicting error types in student programming submissions.
• The semantic embedding improves the accuracy of error prediction, demonstrating its ability to capture meaningful semantic distinctions that syntax alone cannot.
• When integrated into a search-based program repair system, the semantic embedding reduces the search space and improves repair efficiency.
• The model's performance is consistent across diverse programming tasks, including those from the CodeHunt education platform and introductory programming assignments.
• Execution traces as sequential live variable states provide a more natural and effective input for RNNs compared to static syntactic structures.
• The results indicate that semantic-aware program embeddings are more robust and informative for program repair than purely syntactic representations.