[Paper Review] Terminologies augmented recurrent neural network model for clinical named entity recognition

Ivan Lerner, Nicolás Paris+1 more|arXiv (Cornell University)|Apr 25, 2019

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology29 references37 citations

TL;DR

The paper evaluates a terminology-based baseline, a biGRU-CRF model, and a hybrid system that uses terminology predictions as features for biGRU-CRF for clinical NER in English (i2b2-2009) and French (APcNER), showing the hybrid system achieves the best exact-match F1 scores across corpora.

ABSTRACT

We aimed to enhance the performance of a supervised model for clinical named-entity recognition (NER) using medical terminologies. In order to evaluate our system in French, we built a corpus for 5 types of clinical entities. We used a terminology-based system as baseline, built upon UMLS and SNOMED. Then, we evaluated a biGRU-CRF, and an hybrid system using the prediction of the terminology-based system as feature for the biGRU-CRF. In English, we evaluated the NER systems on the i2b2-2009 Medication Challenge for Drug name recognition, which contained 8,573 entities for 268 documents. In French, we built APcNER, a corpus of 147 documents annotated for 5 entities (drug name, sign or symptom, disease or disorder, diagnostic procedure or lab test and therapeutic procedure). We evaluated each NER systems using exact and partial match definition of F-measure for NER. The APcNER contains 4,837 entities which took 28 hours to annotate, the inter-annotator agreement was acceptable for Drug name in exact match (85%) and acceptable for other entity types in non-exact match (>70%). For drug name recognition on both i2b2-2009 and APcNER, the biGRU-CRF performed better that the terminology-based system, with an exact-match F-measure of 91.1% versus 73% and 81.9% versus 75% respectively. Moreover, the hybrid system outperformed the biGRU-CRF, with an exact-match F-measure of 92.2% versus 91.1% (i2b2-2009) and 88.4% versus 81.9% (APcNER). On APcNER corpus, the micro-average F-measure of the hybrid system on the 5 entities was 69.5% in exact match, and 84.1% in non-exact match. APcNER is a French corpus for clinical-NER of five type of entities which covers a large variety of document types. Extending supervised model with terminology allowed for an easy performance gain, especially in low regimes of entities, and established near state of the art results on the i2b2-2009 corpus.

Motivation & Objective

• Enhance supervised clinical NER performance with medical terminologies from UMLS and SNOMED.
• Evaluate NER systems on English (i2b2-2009 Medication Challenge) and French APcNER corpus.
• Compare terminology-based baseline, biGRU-CRF, and a hybrid approach using terminology-derived features.

Proposed method

• Construct terminology-based baseline using UMLS and SNOMED.
• Train a biGRU-CRF for NER on clinical data.
• Develop a hybrid system that uses predictions from the terminology-based system as features for the biGRU-CRF.
• Evaluate using exact and partial (non-exact) match F-measure.
• Annotate APcNER French corpus with five entity types (drug name, sign/symptom, disease/disorder, diagnostic procedure or lab test, therapeutic procedure).

Experimental results

Research questions

• RQ1 How does a terminology-based baseline perform on clinical NER compared to a neural model?
• RQ2 Does incorporating terminology-based predictions as features improve neural NER performance?
• RQ3 What are the NER performance levels on English i2b2-2009 and French APcNER datasets under exact and non-exact matching?

Key findings

• BiGRU-CRF outperforms the terminology-based baseline for drug name recognition on both corpora (i2b2-2009: 91.1% vs 73%; APcNER: 81.9% vs 75%).
• The hybrid system surpasses the biGRU-CRF baseline (i2b2-2009: 92.2% vs 91.1%; APcNER: 88.4% vs 81.9%).
• On APcNER, the hybrid system achieves a micro-average exact-match F-measure of 69.5% and non-exact F-measure of 84.1% for the five entities.
• APcNER corpus is a French clinical-NER dataset with 4,837 entities annotated over 147 documents, with acceptable inter-annotator agreement (exact match for Drug name 85%, other entities >70% non-exact).
• Extending supervised models with terminology yields performance gains, achieving near state-of-the-art results on i2b2-2009.