[Paper Review] Learning to Represent Edits

Pengcheng Yin, Graham Neubig+3 more|arXiv (Cornell University)|Oct 31, 2018

Topic Modeling24 references48 citations

TL;DR

The paper proposes an autoencoder-like framework with an edit encoder fΔ and a neural editor α to learn distributed representations of edits, enabling applying edits to new inputs; experiments on natural language and source code show the representations capture semantic and structural edit patterns.

ABSTRACT

We introduce the problem of learning distributed representations of edits. By combining a "neural editor" with an "edit encoder", our models learn to represent the salient information of an edit and can be used to apply edits to new inputs. We experiment on natural language and source code edit data. Our evaluation yields promising results that suggest that our neural network models learn to capture the structure and semantics of edits. We hope that this interesting task and data source will inspire other researchers to work further on this problem.

Motivation & Objective

• Motivate the problem of learning distributed representations of edits from real edit data.
• Develop an autoencoder-style framework that encodes edits and reconstructs edited inputs.
• Explore architectures that leverage textual and code structure to represent and apply edits.
• Evaluate whether learned edit representations cluster semantically similar edits and transfer edits to new contexts.

Proposed method

• Introduce an edit representation function fΔ that maps an edit x− → x+ to a real-valued vector in R^n with a bottleneck to encourage semantic content.
• Train a neural editor α that uses fΔ(x−, x+) and a new input x−′ to generate x+′, effectively applying the edit in a new context.
• Model Pα as either a sequence-to-sequence editor for text with token-level copying or a graph-to-tree editor for code with a graph neural encoder (GGNN) and a tree-based decoder with subtree copying.
• Construct edit representations via two encodings: a sequence-based diff-augmented encoding and a graph-based encoding that merges x− and x+ into a single edit graph.
• Impose a bottleneck by limiting the dimensionality of fΔ to force the representation to capture only the essential edit information.
• Evaluate end-to-end performance with gold-standard edit representations and in transfer scenarios using a diverse set of datasets.

Experimental results

Research questions

• RQ1 Can semantically equivalent edits be grouped together via learned edit representations?
• RQ2 Can edits learned in one context be transferred to apply the same edit in a different context?
• RQ3 Do edit representations capture meaningful structure in both natural language edits and code edits?
• RQ4 Which architectural choices (sequence vs graph based) best capture edit semantics and enable transfer?
• RQ5 How well do the learned representations support clustering and nearest-neighbor retrieval of edits?

Key findings

• The model learns dense edit representations that form meaningful clusters and reflect edit semantics in qualitative visualizations and quantitative metrics.
• Graph-based edit encoding can capture higher-level structural edit patterns and improve certain transfer scenarios, though sequence-based encoders sometimes perform better on end-to-end prediction when gold edits are provided.
• Neural editors using the edit representations achieve higher accuracy and lower perplexity than bag-of-edits baselines, across GitHubEdits and WikiAtomicEdits datasets.
• End-to-end transfer experiments show that using edit representations from one example can predict edits in other examples within the same fixer category, with best results when using a Graph2Tree editor with sequential edit encoding.
• The approach yields promising evidence that learned edit representations encode semantics of edits and can generalize to unseen contexts.