[Paper Review] PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts

Franck Dernoncourt, Ji Young Lee|arXiv (Cornell University)|Oct 16, 2017

Artificial Intelligence in Healthcare and EducationMedicine73 citations

TL;DR

A large, publicly available dataset (PubMed 200k RCT) of ~195k randomized controlled trial abstracts with labeled sentences for sequential sentence classification, plus baselines.

ABSTRACT

We present PubMed 200k RCT, a new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field.

Motivation & Objective

• Motivate the need for large-scale sequential sentence classification in medical abstracts to aid efficient literature skim and information extraction.
• Describe construction of a structured PubMed-based dataset with sentence-level labels (background, objective, method, result, conclusion).
• Provide dataset statistics, splits, and accessible formats to facilitate benchmarking and model development.

Proposed method

• Construct the dataset from PubMed abstracts labeled as RCTs with 3–9 sections and no 'None' labels.
• Provide data in three splits (train/validation/test) and a smaller PubMed 20k RCT subset for convenience.
• Offer two dataset formats: original text and digits-replaced version for preprocessing.
• Evaluate multiple baselines including LR with n-grams, Forward ANN, CRF with context, and bi-ANN with joint components.
• Report performance using F1-scores to enable direct comparison across methods.

Experimental results

Research questions

• RQ1 How well can sequential sentence classification be performed in medical abstracts when leveraging sentence context?
• RQ2 What baseline methods establish a benchmark for large-scale PubMed-based sequential sentence classification?
• RQ3 How does scale (PubMed 200k vs PubMed 20k) impact baseline performance and method effectiveness.

Key findings

• Bi-ANN achieves the highest F1-scores among the tested baselines on both PubMed 20k and PubMed 200k RCTs (e.g., 90.0 and 91.6, respectively).
• CRF generally performs well, outperforming LR and Forward ANN, with smaller gains as data size grows.
• LR with simple n-gram features underperforms relative to neural baselines, highlighting the benefit of context modeling.
• PubMed 200k RCT is the largest available dataset for this task, consisting of 195,654 abstracts and 2.3 million sentences.
• The authors provide clear baselines to enable direct comparison and benchmarking for future work.