[Paper Review] Self-Attentive Model for Headline Generation

Daniil Gavrilov, Pavel Kalaidin+1 more|arXiv (Cornell University)|Jan 23, 2019

Topic Modeling11 references5 citations

TL;DR

This paper proposes a self-attentive headline generation model based on the Universal Transformer architecture with byte-pair encoding (BPE), achieving state-of-the-art results on the New York Times Annotated corpus (ROUGE-L F1: 24.84) and introducing the RIA corpus for Russian headline generation, where it achieves ROUGE-L F1: 36.81. The model leverages non-local attention to capture long-range dependencies, improving abstractive summarization beyond standard RNN-based approaches.

ABSTRACT

Headline generation is a special type of text summarization task. While the amount of available training data for this task is almost unlimited, it still remains challenging, as learning to generate headlines for news articles implies that the model has strong reasoning about natural language. To overcome this issue, we applied recent Universal Transformer architecture paired with byte-pair encoding technique and achieved new state-of-the-art results on the New York Times Annotated corpus with ROUGE-L F1-score 24.84 and ROUGE-2 F1-score 13.48. We also present the new RIA corpus and reach ROUGE-L F1-score 36.81 and ROUGE-2 F1-score 22.15 on it.

Motivation & Objective

• To improve abstractive headline generation by leveraging the Universal Transformer's ability to model long-range dependencies.
• To address the challenge of generating concise, informative headlines that capture the main idea of news articles.
• To introduce and evaluate on a new Russian-language headline corpus, the RIA corpus, to enable cross-lingual benchmarking.
• To demonstrate that non-local attention mechanisms outperform local RNN-based models in headline generation.
• To evaluate model performance using both automatic metrics (ROUGE) and human evaluation.

Proposed method

• Adopts the Universal Transformer architecture, a modified Transformer that applies iterative refinement to hidden representations, enabling better modeling of long-range dependencies.
• Uses byte-pair encoding (BPE) to tokenize both source and target sequences, improving subword-level representation learning.
• Applies label smoothing during training to improve generalization and reduce overconfidence in predictions.
• Employs an encoder-decoder framework with self-attention mechanisms to attend to relevant parts of the input article during headline generation.
• Uses greedy and beam-search decoding strategies to generate headlines token-by-token, with beam search yielding higher-quality outputs.
• Trains the model end-to-end using maximum likelihood estimation (MLE) to maximize the likelihood of gold-standard headlines.

Experimental results

Research questions

• RQ1 Can the Universal Transformer architecture improve headline generation performance compared to standard RNN-based models?
• RQ2 Does the use of non-local attention mechanisms lead to better capture of long-range dependencies in news articles?
• RQ3 How does the model perform on a newly introduced Russian-language headline corpus (RIA) compared to existing baselines?
• RQ4 To what extent does label smoothing improve generalization and headline quality in abstractive generation?
• RQ5 How close is the model’s output to human-written headlines in human evaluation?

Key findings

• The proposed Universal Transformer model achieves a new state-of-the-art ROUGE-L F1 score of 24.84 on the New York Times Annotated corpus, surpassing prior abstractive models.
• On the newly introduced RIA corpus, the model achieves a ROUGE-L F1 score of 36.81, demonstrating strong performance on a low-resource, non-English language setting.
• Human evaluation shows that the model's headlines are preferred over human-written ones in 42.6% of cases on the NYT corpus and 45.6% on the RIA corpus, indicating strong human-like quality.
• The model outperforms both baseline encoder-decoder and first-sentence baselines across all ROUGE metrics on both datasets.
• Label smoothing improves model generalization, as evidenced by slightly better F1 scores in ablation experiments.
• Despite strong automatic scores, some generated headlines are judged as less reasonable by human annotators, indicating room for improvement in semantic coherence.