[Paper Review] A Retrieve-and-Edit Framework for Predicting Structured Outputs

Tatsunori Hashimoto, Kelvin Guu+2 more|arXiv (Cornell University)|Dec 4, 2018

Software Engineering Research102 citations

TL;DR

The paper proposes a retrieve-and-edit approach that first retrieves a training example based on the input and then edits it to produce a structured output, enabling efficient learning without a hand-crafted metric or joint retriever-editor training.

ABSTRACT

For the task of generating complex outputs such as source code, editing existing outputs can be easier than generating complex outputs from scratch. With this motivation, we propose an approach that first retrieves a training example based on the input (e.g., natural language description) and then edits it to the desired output (e.g., code). Our contribution is a computationally efficient method for learning a retrieval model that embeds the input in a task-dependent way without relying on a hand-crafted metric or incurring the expense of jointly training the retriever with the editor. Our retrieve-and-edit framework can be applied on top of any base model. We show that on a new autocomplete task for GitHub Python code and the Hearthstone cards benchmark, retrieve-and-edit significantly boosts the performance of a vanilla sequence-to-sequence model on both tasks.

Motivation & Objective

• Motivate the difficulty of generating complex outputs from scratch and the potential ease of editing existing outputs.
• Propose a retrieval-based framework that embeds inputs in a task-specific way without hand-crafted metrics.
• Enable a generalizable method that can be applied on top of any base model for structured outputs.

Proposed method

• Retrieve a training example based on the input description.
• Edit the retrieved example to produce the target structured output.
• Learn a retrieval model that embeds inputs in a task-dependent way without jointly training retriever with editor.
• Demonstrate that the framework is model-agnostic and can augment base sequence-to-sequence models.

Experimental results

Research questions

• RQ1 Can retrieve-and-edit improve performance over vanilla sequence-to-sequence models for structured output tasks?
• RQ2 Does a learned, task-dependent retrieval embedding without a handcrafted metric offer practical benefits?
• RQ3 Is the framework effective across different domains and base models?

Key findings

• Retrieve-and-edit significantly boosts performance of a vanilla sequence-to-sequence model on a GitHub Python code autocomplete task.
• Retrieve-and-edit also improves results on the Hearthstone cards benchmark.
• The method provides a computationally efficient way to incorporate retrieval without joint retriever-editor training.