[Paper Review] Two Huge Title and Keyword Generation Corpora of Research Articles

Erion Çano, Ondřej Bojar|arXiv (Cornell University)|Feb 11, 2020

Advanced Text Analysis Techniques48 references4 citations

TL;DR

This paper introduces two large-scale, publicly available corpora—OAGSX (34 million records) for text summarization and OAGKX (23 million records) for keyword generation—derived from the Open Academic Graph. The authors preprocess metadata from research articles and establish performance baselines using extractive and abstractive neural methods, demonstrating that abstractive models outperform extractive ones despite longer training times.

ABSTRACT

Recent developments in sequence-to-sequence learning with neural networks have considerably improved the quality of automatically generated text summaries and document keywords, stipulating the need for even bigger training corpora. Metadata of research articles are usually easy to find online and can be used to perform research on various tasks. In this paper, we introduce two huge datasets for text summarization (OAGSX) and keyword generation (OAGKX) research, containing 34 million and 23 million records, respectively. The data were retrieved from the Open Academic Graph which is a network of research profiles and publications. We carefully processed each record and also tried several extractive and abstractive methods of both tasks to create performance baselines for other researchers. We further illustrate the performance of those methods previewing their outputs. In the near future, we would like to apply topic modeling on the two sets to derive subsets of research articles from more specific disciplines.

Motivation & Objective

• To address the growing need for large-scale training data in neural sequence-to-sequence models for text summarization and keyword generation.
• To create and release two large, high-quality, freely accessible corpora from research article metadata.
• To establish performance benchmarks using both extractive and abstractive methods on representative subsets of the data.
• To enable future research on topic-specific sub-corpora through topic modeling of the full datasets.

Proposed method

• Retrieved metadata (titles, abstracts, keywords) from the Open Academic Graph (OAG), a large-scale academic knowledge network.
• Performed careful data cleaning and preprocessing to ensure consistency and quality across the 34 million OAGSX and 23 million OAGKX records.
• Evaluated multiple extractive and abstractive models: TopicRank, RAKE, Maui (supervised extractive), CopyRNN, and CovRNN (abstractive with coverage and copying mechanisms).
• Used ROUGE scores for text summarization and F1@k for keyword generation to compare model performance on minisets derived from the corpora.
• Applied neural sequence-to-sequence models with attention, coverage, and copying mechanisms to improve abstractive keyword generation.
• Conducted statistical analysis on text lengths and lexical similarities between abstracts, titles, and keywords to characterize the corpora.

Experimental results

Research questions

• RQ1 How do extractive and abstractive neural models compare in performance on large-scale research article title and keyword generation tasks?
• RQ2 What is the impact of model architecture and training time on the quality of generated titles and keywords?
• RQ3 Can large, freely available corpora of research article metadata significantly improve the performance of text summarization and keyword generation systems?
• RQ4 To what extent do abstractive models with copying and coverage mechanisms reduce hallucination and repetition in keyword generation?
• RQ5 Can topic modeling on these corpora yield domain-specific sub-corpora for more focused research?

Key findings

• Abstractive models such as CopyRNN and CovRNN achieved significantly higher F1@7 scores (peak 29.15%) than extractive methods, indicating superior performance in keyword generation.
• The abstractive models, particularly CovRNN, outperformed all extractive methods, including Maui, TopicRank, and RAKE, on the OAGKX benchmark.
• Training abstractive models took approximately three days, whereas extractive methods produced results in minutes, highlighting a trade-off between performance and computational cost.
• The average number of keywords per article was 5.9, explaining the similarity in F1@7 and F1@10 scores across methods.
• Despite grammatical correctness, many generated keywords showed partial or full mismatches with ground-truth keywords, and some models produced novel but incorrect phrases like "mortality of rats".
• The OAGSX and OAGKX corpora are, to the best of the authors' knowledge, the largest freely available datasets of their kind for research article title and keyword generation tasks.