[Paper Review] Bio-YODIE: A Named Entity Linking System for Biomedical Text

Genevieve Gorrell, Xingyi Song+1 more|arXiv (Cornell University)|Nov 12, 2018

Biomedical Text Mining and Ontologies6 references4 citations

TL;DR

Bio-YODIE is a biomedical named entity linking system that uses a 'push' approach by matching text mentions against a pre-processed gazetteer of UMLS concepts, enabling faster and more robust performance than MetaMap. It achieves competitive accuracy and higher recall than MetaMap and MetaMapLite, with a 59.3% F1 score on the MIMIC-II test set, demonstrating that simpler, gazetteer-based systems can outperform complex NER-driven approaches in biomedical NEL.

ABSTRACT

Ever-expanding volumes of biomedical text require automated semantic annotation techniques to curate and put to best use. An established field of research seeks to link mentions in text to knowledge bases such as those included in the UMLS (Unified Medical Language System), in order to enable a more sophisticated understanding. This work has yielded good results for tasks such as curating literature, but increasingly, annotation systems are more broadly applied. Medical vocabularies are expanding in size, and with them the extent of term ambiguity. Document collections are increasing in size and complexity, creating a greater need for speed and robustness. Furthermore, as the technologies are turned to new tasks, requirements change; for example greater coverage of expressions may be required in order to annotate patient records, and greater accuracy may be needed for applications that affect patients. This places new demands on the approaches currently in use. In this work, we present a new system, Bio-YODIE, and compare it to two other popular systems in order to give guidance about suitable approaches in different scenarios and how systems might be designed to accommodate future needs.

Motivation & Objective

• To address the growing need for efficient and accurate named entity linking (NEL) in biomedical text due to increasing volumes of clinical and research literature.
• To evaluate and compare the performance of different NEL approaches—particularly 'pull' (MetaMap) versus 'push' (MetaMapLite, Bio-YODIE) paradigms—in biomedical contexts.
• To investigate whether simpler, gazetteer-based systems can achieve higher recall and competitive F1 scores without sacrificing precision, especially in complex clinical text.
• To provide guidance on system selection based on application-specific needs such as speed, coverage, and accuracy.
• To highlight the importance of evaluating NEL systems on diverse, real-world corpora like MIMIC-II, which include complex, informal clinical notes.

Proposed method

• Bio-YODIE employs a 'push' approach by pre-processing and expanding a UMLS-based gazetteer with linguistic permutations (e.g., pluralization, inflection) to increase match coverage.
• It uses a string-matching pipeline to efficiently search for these pre-processed terms in biomedical text, reducing runtime complexity compared to NER-based systems.
• The system leverages the GATE natural language processing framework for integration and text processing, focusing on speed and scalability.
• Disambiguation is handled through context-aware matching and semantic type filtering, though not via complex linguistic parsing as in MetaMap.
• Performance is evaluated using the MIMIC-II corpus, which contains manually annotated patient notes with UMLS CUIs across multiple semantic types.
• Metrics include precision, recall, F1, accuracy, and Scott’s Pi for inter-annotator agreement, enabling robust comparison across MetaMap, MetaMapLite, and Bio-YODIE.

Experimental results

Research questions

• RQ1 How does the performance of a gazetteer-based NEL system like Bio-YODIE compare to a linguistically complex 'pull' system like MetaMap in terms of speed, recall, and F1 on clinical text?
• RQ2 Can a simpler, 'push' approach achieve higher recall than MetaMap while maintaining or improving precision, especially in the presence of term variations and informal language?
• RQ3 To what extent do coverage extension strategies—such as pre-computed linguistic permutations—affect the performance of biomedical NEL systems?
• RQ4 How do different NEL systems handle ambiguity in clinical text, particularly for terms with multiple possible CUI mappings?
• RQ5 Does the use of a gold-standard corpus with multiple valid CUIs per mention improve the evaluation of NEL systems compared to single-CUI annotation?

Key findings

• Bio-YODIE achieved a 59.3% F1 score on the MIMIC-II test set, outperforming MetaMap (57.1%) and matching MetaMapLite (59.7%), indicating strong overall performance.
• Bio-YODIE recorded the highest recall (60.5%) among the three systems, demonstrating superior coverage of relevant entities in clinical text.
• MetaMapLite achieved the highest precision (65.4%), indicating fewer false positives, while Bio-YODIE balanced recall and precision effectively.
• Bio-YODIE processed text in 573 seconds, significantly faster than MetaMap’s 3811 seconds, confirming the efficiency gain of the 'push' approach.
• Scott’s Pi values of 0.882 for Bio-YODIE and 0.876 for MetaMapLite indicate high inter-annotator agreement, supporting the reliability of the evaluation.
• The results suggest that 'push' systems like Bio-YODIE and MetaMapLite can outperform complex 'pull' systems like MetaMap in both speed and coverage, especially in real-world clinical text.