[Paper Review] Comprehensive Named Entity Recognition on CORD-19 with Distant or Weak Supervision

Xuan Wang, Xiangchen Song+3 more|arXiv (Cornell University)|Mar 27, 2020

Topic Modeling4 references38 citations

TL;DR

CORD-NER creates a comprehensive 75-type NER dataset for the CORD-19 corpus using distantly and weakly supervised methods, outperforming SciSpacy on key biomedical types and allowing incremental addition of new documents and entity types.

ABSTRACT

We created this CORD-NER dataset with comprehensive named entity recognition (NER) on the COVID-19 Open Research Dataset Challenge (CORD-19) corpus (2020-03-13). This CORD-NER dataset covers 75 fine-grained entity types: In addition to the common biomedical entity types (e.g., genes, chemicals and diseases), it covers many new entity types related explicitly to the COVID-19 studies (e.g., coronaviruses, viral proteins, evolution, materials, substrates and immune responses), which may benefit research on COVID-19 related virus, spreading mechanisms, and potential vaccines. CORD-NER annotation is a combination of four sources with different NER methods. The quality of CORD-NER annotation surpasses SciSpacy (over 10% higher on the F1 score based on a sample set of documents), a fully supervised BioNER tool. Moreover, CORD-NER supports incrementally adding new documents as well as adding new entity types when needed by adding dozens of seeds as the input examples. We will constantly update CORD-NER based on the incremental updates of the CORD-19 corpus and the improvement of our system.

Motivation & Objective

• Address the need for rapidly adaptable NER for COVID-19 related content without extensive manual annotation.
• Create a comprehensive 75-type NER schema for the CORD-19 corpus including COVID-19 specific types.
• Combine multiple NER sources and supervision signals to maximize annotation quality.
• Enable incremental addition of new documents and new entity types with seed-based guidance.

Proposed method

• Construct the CORD-NER corpus from 29,500 CORD-19 documents by merging metadata with full text and tokenizing via AutoPhrase followed by Spacy.
• Fuse four NER sources: (i) pre-trained general NER from Spacy, (ii) pre-trained biomedical NER from SciSpacy, (iii) KB-guided distantly supervised NER using UMLS as the knowledge base, (iv) seed-guided weakly supervised NER for COVID-19 specific types with seed expansion.
• Align and organize entity types from the four sources into a unified 75-item CORD-NER hierarchy (mapping SciSpacy types to UMLS and pruning for corpus relevance).
• Perform NER annotation across all 75 types with the four methods, then merge results with conflict resolution prioritizing method-based annotation quality.
• Produce consolidated outputs (CORD-NER.json and CORD-NER-full.json) containing metadata, corpus, and NER results, and support incremental updates to the corpus and types.

Experimental results

Research questions

• RQ1 How well does the proposed CORD-NER system perform on broad biomedical entity types (gene, chemical, disease) compared to existing NER tools?
• RQ2 Can distantly supervised and seed-guided weak supervision effectively recognize COVID-19–specific entity types with minimal human labeling?
• RQ3 Does incorporating AutoPhrase-preserved phrase constructs improve distant/weak supervision NER performance on CORD-19?
• RQ4 Is the system capable of incremental updates by adding new documents and new entity types with limited seed examples?
• RQ5 What qualitative benefits does CORD-NER provide for downstream COVID-19 text mining tasks?

Key findings

• CORD-NER surpasses SciSpacy on chemical and disease entity types by a notable margin in F1 score (over 10% in their evaluation sample).
• The system enables recognition of COVID-19–specific concepts (e.g., coronaviruses, viral proteins, immune responses) beyond standard biomedical types.
• The approach can operate with distantly supervised and seed-guided weak supervision without requiring human-annotated training data for the NER models.
• CAND-NER demonstrates higher recall/precision trade-offs for target types than some fully supervised baselines in their reported comparisons.
• The framework supports incremental addition of documents and new entity types by supplying dozens of seed examples.