[论文解读] PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts

Franck Dernoncourt, Ji Young Lee|arXiv (Cornell University)|Oct 16, 2017

Artificial Intelligence in Healthcare and Education被引用 73

一句话总结

一个大型、公开可用的数据集（PubMed 200k RCT），包含约195k个随机对照试验摘要，带有用于序列句子分类的标注句子，以及基线。

ABSTRACT

We present PubMed 200k RCT, a new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field.

研究动机与目标

• 为大型序列句子分类在医学摘要中的需求提供动机，以帮助高效的文献浏览和信息提取。
• 描述基于PubMed构建的带有句子级标签（背景、目标、方法、结果、结论）的结构化数据集的构建。
• 提供数据集统计信息、划分和可访问格式，以便基准测试和模型开发。

提出的方法

• 从被标注为RCT且具有3–9个章节且没有“None”标签的PubMed摘要构建数据集。
• 提供三种划分（训练/验证/测试）以及一个用于方便的小型PubMed 20k RCT子集。
• 提供两种数据集格式：原始文本和用于预处理的数字替换版本。
• 评估多种基线方法，包括带n-gram的LR、前向ANN、带上下文的CRF，以及带联合组件的bi-ANN。
• 使用F1分数报告性能，以便在方法之间实现直接比较。

实验结果

研究问题

• RQ1 在利用句子上下文时，医学摘要的序列句子分类在多大程度上可以实现？
• RQ2 哪些基线方法可以为基于PubMed的大规模序列句子分类设定基准？
• RQ3 规模（PubMed 200k vs PubMed 20k）如何影响基线性能和方法有效性？

主要发现

• Bi-ANN在PubMed 20k和PubMed 200k RCT上都取得了测试基线中的最高F1分数（例如，分别为90.0和91.6）。
• CRF通常表现良好，优于LR和Forward ANN，数据规模增大时增益较小。
• LR with simple n-gram features underperforms relative to neural baselines, highlighting the benefit of context modeling.
• PubMed 200k RCT is the largest available dataset for this task, consisting of 195,654 abstracts and 2.3 million sentences.
• The authors provide clear baselines to enable direct comparison and benchmarking for future work.