[Paper Review] Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization

Jun Suzuki, Masaaki Nagata|arXiv (Cornell University)|Dec 31, 2016

Natural Language Processing Techniques18 references4 citations

TL;DR

This paper proposes a word-frequency estimation (WFE) sub-model that jointly estimates the upper-bound frequency of each target vocabulary word during encoding and uses this to suppress redundant repeating generations in RNN-based abstractive summarization. By controlling output word generation based on estimated frequency limits, the method significantly reduces repetition and improves ROUGE scores, achieving state-of-the-art performance on benchmark datasets without changing the main model architecture.

ABSTRACT

This paper tackles the reduction of redundant repeating generation that is often observed in RNN-based encoder-decoder models. Our basic idea is to jointly estimate the upper-bound frequency of each target vocabulary in the encoder and control the output words based on the estimation in the decoder. Our method shows significant improvement over a strong RNN-based encoder-decoder baseline and achieved its best results on an abstractive summarization benchmark.

Motivation & Objective

• To address the problem of redundant repeating generation in RNN-based encoder-decoder models for abstractive summarization.
• To improve performance on abstractive summarization tasks where output length is constrained and redundancy wastes limited space.
• To develop a method that explicitly manages word frequency in the output to prevent over-generation of specific words or phrases.
• To evaluate the effectiveness of the proposed method on standard abstractive summarization benchmarks, including Gigaword and DUC-2004.
• To demonstrate that frequency-based control can improve both automatic evaluation metrics and human-rated generation quality.

Proposed method

• The method introduces a word-frequency estimation (WFE) sub-model that estimates the maximum number of times each target word can appear in the summary during the encoding phase.
• The WFE sub-model tracks how many times each word has been generated so far and predicts its potential future usage, enabling suppression of over-generation.
• During decoding, the model uses the estimated frequency bounds to penalize or suppress output tokens that would exceed their estimated upper limit.
• The approach integrates the WFE sub-model into a standard RNN-based encoder-decoder with global attention, without modifying the main architecture.
• The WFE component uses a learnable function to estimate word frequencies and applies a margin-based constraint to avoid overfitting to exact counts.
• The method is trained via standard point-wise log-likelihood maximization and evaluated using beam search decoding with K-best selection.

Experimental results

Research questions

• RQ1 Can a sub-model that estimates the upper-bound frequency of each target word reduce redundant repeating generations in abstractive summarization?
• RQ2 Does frequency-based control improve automatic evaluation metrics such as ROUGE scores in low-resource or constrained-length summarization tasks?
• RQ3 Can the proposed method outperform strong RNN-based baselines without architectural changes or complex training schemes?
• RQ4 How effective is the WFE sub-model in practice, even if its frequency estimation accuracy is suboptimal?
• RQ5 Can the method be generalized to other lossy compression tasks like image captioning?

Key findings

• The EncDec+WFE model significantly outperformed the strong baseline on ROUGE-1(F), ROUGE-2(F), and ROUGE-L(F) scores for the Gigaword dataset, with improvements of 33.65, 16.12, and 31.37 respectively.
• On the DUC-2004 benchmark, EncDec+WFE achieved state-of-the-art results, outperforming previous top systems including those using minimum risk training (MRT).
• Qualitative analysis showed that the method effectively reduced repeating phrases and words, such as 'black women black women' and 'pinochet’s pinochet’s', which were common in the baseline.
• Despite the WFE sub-model’s moderate frequency estimation accuracy (as shown in the confusion matrix), it still provided substantial gains in overall generation quality and ROUGE scores.
• The results suggest that even imprecise frequency estimation can be highly effective when used to control generation, indicating the robustness of the approach.
• The method is compatible with sequence-level training objectives like MRT, suggesting potential for further performance gains when combined with such techniques.