[Paper Review] Tree2Tree Neural Translation Model for Learning Source Code Changes.

Saikat Chakraborty, Miltiadis Allamanis+1 more|arXiv (Cornell University)|Sep 30, 2018

Software Engineering ResearchComputer Science54 references21 citations

TL;DR

This paper proposes CODIT, a tree-based neural machine translation model that learns source code change patterns from real-world patches by modeling code as abstract syntax trees (ASTs). Trained on 24,000 changes and evaluated on 5,000 patches, CODIT achieves strong performance in suggesting accurate code changes and fixes 25 out of 80 bugs in Defects4J, demonstrating its effectiveness in learning syntactically valid, reusable code transformations.

ABSTRACT

The way developers edit day-to-day code tends to be repetitive, often using existing code elements. Many researchers have tried to automate repetitive code changes by learning from specific change templates which are applied to limited scope. The advancement of deep neural networks and the availability of vast open-source evolutionary data opens up the possibility of automatically learning those templates from the wild. However, deep neural network based modeling for code changes and code in general introduces some specific problems that needs specific attention from research community. For instance, compared to natural language, source code vocabulary can be significantly larger. Further, good changes in code do not break its syntactic structure. Thus, deploying state-of-the-art neural network models without adapting the methods to the source code domain yields sub-optimal results. To this end, we propose a novel tree-based neural network system to model source code changes and learn code change patterns from the wild. Specifically, we propose a tree-based neural machine translation model to learn the probability distribution of changes in code. We realize our model with a change suggestion engine, CODIT, and train the model with more than 24k real-world changes and evaluate it on 5k patches. Our evaluation shows the effectiveness of CODITin learning and suggesting patches. CODIT can also learn specific bug fix pattern from bug fixing patches and can fix 25 bugs out of 80 bugs in Defects4J.

Motivation & Objective

• To address the challenge of automating repetitive code changes by learning patterns from real-world code evolution data.
• To overcome limitations of standard neural networks in code translation by modeling code as trees to preserve syntactic structure.
• To develop a system that learns and suggests semantically and syntactically valid code changes without relying on predefined templates.
• To evaluate the model’s ability to generalize to real-world bug-fixing scenarios using standard benchmarks.

Proposed method

• The model uses a tree-to-tree neural machine translation framework to map source code ASTs to modified ASTs, preserving syntactic structure.
• It employs an encoder-decoder architecture with tree-structured long short-term memory (Tree-LSTM) networks to encode and decode ASTs.
• The model is trained end-to-end on 24,000 real-world code changes extracted from open-source repositories.
• A change suggestion engine, CODIT, applies the trained model to generate patch suggestions during code editing.
• The model learns from evolutionary data, capturing common refactoring and bug-fixing patterns without hard-coded templates.
• Evaluation is performed on 5,000 real patches, including 80 Defects4J bug-fixing examples.

Experimental results

Research questions

• RQ1 Can a neural model learn meaningful and syntactically valid code change patterns directly from real-world code evolution data?
• RQ2 How well does the tree-based neural translation model generalize to unseen code changes compared to template-based or sequence-based models?
• RQ3 To what extent can the model learn and apply specific bug-fixing patterns from patches in standard benchmarks?
• RQ4 Can the model suggest accurate and syntactically correct code changes in real-world development scenarios?

Key findings

• CODIT successfully learns and suggests code changes with high syntactic fidelity by modeling code as abstract syntax trees.
• The model achieves strong performance on a held-out test set of 5,000 patches, demonstrating generalization to unseen changes.
• CODIT fixes 25 out of 80 bugs in the Defects4J benchmark, showing its effectiveness in real-world bug-fixing scenarios.
• The tree-based architecture enables better preservation of code structure compared to standard sequence-based neural models.
• The model learns reusable change patterns from raw code evolution data without requiring manual template curation.