[Paper Review] LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents

Debanjan Mahata, Naveen Agarwal+6 more|arXiv (Cornell University)|Mar 29, 2022

Advanced Text Analysis Techniques6 citations

TL;DR

This paper introduces LDKP, a large-scale dataset comprising approximately 1.3 million and 100,000 long scientific documents with full-text content and rich metadata, designed to address the limitations of existing keyphrase extraction (KPE) and keyphrase generation (KPG) datasets that rely on short titles and abstracts. By mapping existing KP20K and OAGKx keyphrase datasets to full-text papers from the S2ORC corpus, LDKP increases average document length from ~8 to over 190 sentences and boosts the proportion of keyphrases present in the input text from ~53% to up to 76%, enabling more realistic evaluation of KPE/KPG models on long, real-world documents.

ABSTRACT

Identifying keyphrases (KPs) from text documents is a fundamental task in natural language processing and information retrieval. Vast majority of the benchmark datasets for this task are from the scientific domain containing only the document title and abstract information. This limits keyphrase extraction (KPE) and keyphrase generation (KPG) algorithms to identify keyphrases from human-written summaries that are often very short (approx 8 sentences). This presents three challenges for real-world applications: human-written summaries are unavailable for most documents, the documents are almost always long, and a high percentage of KPs are directly found beyond the limited context of title and abstract. Therefore, we release two extensive corpora mapping KPs of ~1.3M and ~100K scientific articles with their fully extracted text and additional metadata including publication venue, year, author, field of study, and citations for facilitating research on this real-world problem.

Motivation & Objective

• Address the critical gap in keyphrase extraction (KPE) and keyphrase generation (KPG) research caused by reliance on short, human-written abstracts instead of full scientific documents.
• Overcome the limitations of existing datasets that restrict keyphrase identification to only 8–10 sentences, which does not reflect real-world document length and content distribution.
• Provide a scalable, publicly available dataset with full-text scientific articles and associated keyphrases to support training and evaluation of modern deep learning models.
• Facilitate research on long-document keyphrase extraction by including rich metadata such as publication venue, year, authors, citations, and citation contexts.
• Enable evaluation of KPE/KPG models on long documents with a higher proportion of keyphrases present in the input text—up to 76.11% in LDKP3K—compared to existing datasets (~50–57%).

Proposed method

• Mapped existing keyphrase datasets (KP20K and OAGKx) to full scientific articles from the S2ORC corpus using paper titles as a linking key.
• Standardized section headers (e.g., 'Introduction', 'Methodology') across all documents by replacing variant forms (e.g., '1. Introduction', 'I. Introduction') with a unified format.
• Filtered out low-quality keyphrases such as unigrams, numeric-only entries, and incorrectly parsed phrases (e.g., '2,4- dichlorophenoxyacetic acid' split into multiple tokens).
• Created three data splits (small, medium, large) for both LDKP3K (100K documents) and LDKP10K (1.3M documents) to support training on resource-constrained platforms.
• Preserved field-of-study, citation, and metadata information (e.g., venue, year, DOI, PubMed ID) to enrich the dataset for downstream research.
• Ensured balanced distribution of fields of study across train, validation, and test splits through careful sampling and manual verification of title conflicts.

Experimental results

Research questions

• RQ1 To what extent do existing KPE and KPG models generalize when evaluated on long scientific documents instead of short abstracts?
• RQ2 How does increasing the average document length from ~8 to over 190 sentences affect the proportion of keyphrases present in the input text?
• RQ3 Can models trained on full-text documents with rich metadata (e.g., citations, venue, year) achieve better keyphrase extraction performance than those trained on abstract-only data?
• RQ4 What is the impact of document length and keyphrase distribution on the performance of modern deep learning models in keyphrase extraction and generation?
• RQ5 How do different data splits (small, medium, large) affect the reproducibility and scalability of KPE/KPG research on large-scale datasets?

Key findings

• The LDKP3K dataset contains 100,000 full scientific articles with an average of 280.67 sentences per document, a 30-fold increase compared to the average of 8.87 sentences in existing datasets.
• The LDKP10K dataset comprises 1.3 million full scientific articles with an average of 194.76 sentences per document, significantly expanding the scale of available long-document KPE data.
• The proportion of keyphrases found within the input text increased from 52.7% (OAGKx) to 63.65% in LDKP10K and from 57.4% (KP20K) to 76.11% in LDKP3K, indicating a substantial improvement in input coverage.
• The LDKP datasets include rich metadata such as publication venue, year, authors, citations, and citation contexts, enabling more nuanced downstream analysis and model training.
• The datasets are available in multiple sizes (small, medium, large) to support training on low-resource platforms like Google Colab, enhancing accessibility for researchers with limited computational resources.
• Field-of-study distributions across train, validation, and test splits are well-balanced, as confirmed by visualizations in Figures 1 and 2, ensuring reliable model evaluation.