[Paper Review] Incorporating Copying Mechanism in Sequence-to-Sequence Learning

Jiatao Gu, Zhengdong Lu+2 more|arXiv (Cornell University)|Mar 21, 2016

Topic ModelingComputer Science15 references258 citations

TL;DR

CopyNet extends seq2seq with a differentiable copying mechanism that can copy subsequences from the input, improving performance on tasks like synthetic pattern learning, text summarization, and single-turn dialogue."

ABSTRACT

We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks.

Motivation & Objective

• Motivate the need for accurate copying of input subsequences in Seq2Seq tasks (e.g., entity names, dates).
• Propose a unified encoder–decoder model (CopyNet) that integrates both generation and copying in a differentiable framework.
• Demonstrate CopyNet’s effectiveness across synthetic, summarization, and dialogue datasets.
• Show that copying improves handling of out-of-vocabulary words by leveraging source-side content.

Proposed method

• Introduce CopyNet with an encoder–decoder architecture and a mixed prediction model that combines generate-mode and copy-mode.
• Define copy-mode scores using source hidden states to select input subsequences for copying (Eq. 6).
• Define generate-mode scores using standard decoder output with parameterized vocab scores (Eq. 7).
• Compute a shared normalization Z across both modes to form a softmax-based competition between generate and copy (Eq. 4–6).
• Incorporate a hybrid addressing mechanism with attentive read (content-based) and selective read (location-based) of the source memory M (Sections 3.3–3.4).
• Update decoder state to include previous word embedding and a location-aware selective read vector to guide subsequent steps (Eq. 9).
• Train end-to-end by minimizing negative log-likelihood without extra mode labels (Eq. 10).

Experimental results

Research questions

• RQ1 Can a differentiable copying mechanism improve Seq2Seq models on tasks requiring faithful reproduction of input segments?
• RQ2 How does CopyNet balance copying and generating, and how does it handle OOV words through source copying?
• RQ3 Does CopyNet improve performance across synthetic, summarization, and dialogue datasets compared with standard encoder–decoder models with/without attention?
• RQ4 What is the role of hybrid (content- and location-based) addressing in enabling effective copying?

Key findings

• CopyNet significantly outperforms standard Enc-Dec and RNNSearch on synthetic copying tasks (Table 1).
• On LCSTS Chinese summarization, CopyNet achieves higher ROUGE scores than baselines, with +C and +W variants showing strong gains (Table 3).
• In single-turn dialogue, CopyNet yields higher Top-1 and Top-10 decoding accuracy than RNNSearch, especially when test data has disjoint training substrings (Table 4).
• CopyNet can copy long OOV subsequences from the source, mitigating the open-vocabulary problem in abstractive tasks (text summarization and dialogue cases).
• The model demonstrates precise coordination between copy-mode and generate-mode, often copying consecutive input segments and inserting generated content to form fluent outputs (case studies in figures).