[Paper Review] An Empirical Study on Neural Keyphrase Generation

Rui Meng, Xingdi Yuan+4 more|arXiv (Cornell University)|Sep 22, 2020

Advanced Text Analysis Techniques37 references4 citations

TL;DR

This empirical study investigates key factors affecting generalization in neural keyphrase generation (KPG) using both One2One and One2Seq training paradigms with RNN and Transformer architectures. The authors find that training order of keyphrases (e.g., present-before-absent) significantly impacts performance, especially for absent keyphrases, and that larger training data improves generalization when properly utilized, offering practical insights for future KPG research.

ABSTRACT

Recent years have seen a flourishing of neural keyphrase generation (KPG) works, including the release of several large-scale datasets and a host of new models to tackle them. Model performance on KPG tasks has increased significantly with evolving deep learning research. However, there lacks a comprehensive comparison among different model designs, and a thorough investigation on related factors that may affect a KPG system's generalization performance. In this empirical study, we aim to fill this gap by providing extensive experimental results and analyzing the most crucial factors impacting the generalizability of KPG models. We hope this study can help clarify some of the uncertainties surrounding the KPG task and facilitate future research on this topic.

Motivation & Objective

• To address the lack of comprehensive comparison among model designs and factors affecting generalization in neural keyphrase generation (KPG).
• To investigate how training paradigms (One2One vs. One2Seq), model architectures (RNN vs. Transformer), and data scale influence KPG system performance.
• To analyze the impact of keyphrase ordering during training on model generalization, particularly for absent keyphrases.
• To provide empirical guidelines for future KPG research by evaluating model behavior across diverse testing distributions and hyperparameter settings.

Proposed method

• The study evaluates RNN and Transformer-based models under both One2One and One2Seq training paradigms on the KP20k dataset.
• One2Seq concatenates all ground-truth keyphrases into a single sequence using <bos>, <sep>, and <eos> tokens, with ordering strategies including Pres-Abs (present first, then absent), Abs-Pres (absent first), and random ordering.
• The RNN model uses GRU-based encoders and decoders with copy and coverage mechanisms, while the Transformer variant replaces GRUs with self-attention blocks.
• Models are trained and evaluated using standard metrics: F1@O, R@50, and F1@10, with early stopping based on validation F1@O scores.
• Multiple training orders are systematically compared to assess their impact on performance across present and absent keyphrase generation.
• Hyperparameter tuning and data augmentation strategies are applied to evaluate the benefits of larger training data.

Experimental results

Research questions

• RQ1 How well do KPG models generalize across different testing distributions, particularly for absent keyphrases?
• RQ2 Does the order of keyphrase targets during training (e.g., present-before-absent vs. absent-before-present) affect model generalization?
• RQ3 To what extent does increasing training data size improve model performance, and how can it be better leveraged?

Key findings

• The Pres-Abs ordering strategy (present keyphrases first, then absent) consistently outperforms other orderings, especially for absent keyphrase generation, achieving 35.9 F1@O on KP20k.
• Models trained with the Abs-Pres ordering (absent first) show significantly worse performance on absent keyphrases, with F1@O dropping to 30.7, indicating that training order has a strong impact on generalization.
• The Transformer architecture (TRANS) outperforms the RNN model across all settings, achieving a peak F1@O of 35.9 on KP20k with the Pres-Abs ordering.
• Larger training data improves generalization: models trained on full data (including augmented data) achieve higher F1@O and R@50 scores than those trained on smaller subsets.
• The number of unique keyphrases generated during inference is highest under the Pres-Abs strategy, indicating better coverage and diversity in predictions.
• Random and Alpha orderings yield the lowest performance, with F1@O values below 25.0, suggesting that arbitrary ordering harms model learning and generalization.