[Paper Review] Enhancing Label Consistency on Document-level Named Entity Recognition

Minbyul Jeong, Jaewoo Kang|arXiv (Cornell University)|Oct 24, 2022

Topic Modeling4 citations

TL;DR

This paper proposes ConNER, a method that enhances label consistency in document-level biomedical named entity recognition by refining labels of modifiers (e.g., adjectives, prepositions) through label refinement and representation distillation. It achieves 7.5–8.6% absolute F1 score improvements on low-consistency datasets by improving label dependency of modifiers without altering the main classification head.

ABSTRACT

Named entity recognition (NER) is a fundamental part of extracting information from documents in biomedical applications. A notable advantage of NER is its consistency in extracting biomedical entities in a document context. Although existing document NER models show consistent predictions, they still do not meet our expectations. We investigated whether the adjectives and prepositions within an entity cause a low label consistency, which results in inconsistent predictions. In this paper, we present our method, ConNER, which enhances the label dependency of modifiers (e.g., adjectives and prepositions) to achieve higher label agreement. ConNER refines the draft labels of the modifiers to improve the output representations of biomedical entities. The effectiveness of our method is demonstrated on four popular biomedical NER datasets; in particular, its efficacy is proved on two datasets with 7.5-8.6% absolute improvements in the F1 score. We interpret that our ConNER method is effective on datasets that have intrinsically low label consistency. In the qualitative analysis, we demonstrate how our approach makes the NER model generate consistent predictions. Our code and resources are available at https://github.com/dmis-lab/ConNER/.

Motivation & Objective

• To address inconsistent predictions in document-level NER models, especially for modifiers like adjectives and prepositions that exhibit low label consistency.
• To improve label dependency of modifiers within biomedical entities to enhance overall prediction consistency.
• To develop a method that strengthens label representations of modifiers without modifying the main classification head.
• To evaluate the method on low-resource and low-consistency biomedical NER benchmarks, demonstrating robustness across diverse datasets.

Proposed method

• ConNER applies a label refinement process to correct draft predictions of modifier tokens (e.g., 'primary', 'hereditary') based on context-aware representations.
• It introduces a distillation loss ($L_{distill}$) that encourages the model to learn consistent label representations by aligning predictions with refined labels.
• The method uses a consistency score ($\phi_{tCon}$) to measure token-level label agreement across training and test sets, guiding refinement on low-consistency tokens.
• It leverages pre-trained biomedical language models (BioBERT, BioLM) as the backbone and applies refinement only on modifier tokens to preserve main classification performance.
• The label refinement is applied during training using a hyperparameter $\Gamma$ to control the strength of refinement, with $\Gamma=0.3$ yielding stable performance across datasets.
• The approach is designed to be plug-and-play, requiring no changes to the main classification layer, making it compatible with existing document NER frameworks.

Experimental results

Research questions

• RQ1 What causes inconsistent predictions in document-level NER models, particularly for modifiers like adjectives and prepositions?
• RQ2 To what extent can improving label dependency of modifiers enhance overall label consistency and F1 performance in biomedical NER?
• RQ3 How does the proposed label refinement and distillation mechanism affect prediction consistency on low-consistency datasets?
• RQ4 Can ConNER generalize to out-of-density tokens and subtokens with low label agreement?
• RQ5 How does the hyperparameter $\Gamma$ influence the stability and performance of the label refinement process?

Key findings

• ConNER achieves 7.5–8.6% absolute F1 score improvements on two low-consistency biomedical NER datasets (NCBI-disease and Gellus), demonstrating strong efficacy on datasets with intrinsic label inconsistency.
• On the NCBI-disease dataset, ConNER increases the consistency score ($\phi_{tCon}$) for 'primary' from 0% (BioLM) to 100% on the test set, showing effective correction of inconsistent predictions.
• For the 'surgical' token, which had a 92.3% consistency score in training, ConNER maintains high consistency in predictions, confirming its ability to preserve label dependency for high-frequency modifiers.
• The method achieves stable performance across five random seeds, with $\Gamma=0.3$ providing consistent gains across all benchmarks, indicating robustness to initialization.
• ConNER successfully predicts 'rectal carcinoma' as a consistent entity, while the baseline model inconsistently labels 'rectal' as entity in one instance and non-entity in another.
• Despite improvements, ConNER struggles with out-of-density tokens like 'mesorectum' and 'margin', and subtokens like 'lymph' with low consistency scores (0.62), indicating limitations in generalization to rare or inconsistent subtokens.