[Paper Review] MedMentions: A Large Biomedical Corpus Annotated with UMLS Concepts

Sunil Mohan, Donghui Li|arXiv (Cornell University)|Feb 25, 2019

Biomedical Text Mining and Ontologies14 references47 citations

TL;DR

MedMentions presents a large manually annotated biomedical corpus linking mentions to UMLS concepts, including a ST21pv sub-corpus and a baseline end-to-end entity recognition/linking model.

ABSTRACT

This paper presents the formal release of MedMentions, a new manually annotated resource for the recognition of biomedical concepts. What distinguishes MedMentions from other annotated biomedical corpora is its size (over 4,000 abstracts and over 350,000 linked mentions), as well as the size of the concept ontology (over 3 million concepts from UMLS 2017) and its broad coverage of biomedical disciplines. In addition to the full corpus, a sub-corpus of MedMentions is also presented, comprising annotations for a subset of UMLS 2017 targeted towards document retrieval. To encourage research in Biomedical Named Entity Recognition and Linking, data splits for training and testing are included in the release, and a baseline model and its metrics for entity linking are also described.

Motivation & Objective

• Provide a large, richly annotated biomedical corpus with broad coverage using UMLS as the target ontology.
• Enable development of end-to-end biomedical concept recognition and linking, including zero-shot scenarios due to vast ontology size.
• Offer a ST21pv sub-corpus tailored for document retrieval and indexing tasks.
• Present training/validation/test splits and baseline metrics to benchmark future concept recognition models.

Proposed method

• Randomly select 5,000 PubMed abstracts from 2016–2017 and finalize 4,392 English biomedical abstracts for annotation.
• Annotate every biomedical mention with the most specific UMLS concept using the 2017 AA release and GATE tooling, ensuring non-overlapping mentions.
• Introduce ST21pv as a filtered 21 semantic-types subset of UMLS for targeted document retrieval.
• Compute corpus statistics and provide 60/20/20 train/dev/test splits with concept overlap analyses.
• Train a baseline end-to-end concept recognition and linking model (TaggerOne) on ST21pv data and report mention- and document-level metrics.

Experimental results

Research questions

• RQ1 What is the scale and coverage of MedMentions and its ST21pv subset when mapped to UMLS concepts?
• RQ2 How does a baseline end-to-end concept recognition and linking model perform on MedMentions at both mention and document levels?
• RQ3 What is the overlap of concepts between training, development, and test partitions, and how many unseen labels appear in test?
• RQ4 How does MedMentions compare to other large biomedical corpora in terms of size, ontology coverage, and annotation granularity?

Key findings

• MedMentions contains 4,392 abstracts with 352,496 total mentions linked to 34,724 unique concepts.
• The ST21pv sub-corpus covers 2,327,250 concepts and 203,282 mentions after filtering.
• The ST21pv test set includes 60%–20%–20% splits with 42% of test concepts unseen in training and 38% unseen in both training and dev.
• A baseline TaggerOne model achieved 0.471 precision, 0.436 recall, and 0.453 F1 at mention level, and 0.536 precision, 0.561 recall, and 0.548 F1 at document level on MM-ST21pv test data.