[Paper Review] A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?

Hongyu Lin, Yaojie Lu+5 more|arXiv (Cornell University)|Apr 25, 2020

Topic Modeling37 references4 citations

TL;DR

This paper investigates whether fine-tuned pretrained models generalize well in open-world NER settings by erasing name regularity, mention coverage, and context diversity from standard benchmarks via randomization tests. Results show name regularity is crucial for generalization, high mention coverage harms it, and extensive training data is redundant for learning context patterns.

ABSTRACT

Fine-tuning pretrained model has achieved promising performance on standard NER benchmarks. Generally, these benchmarks are blessed with strong name regularity, high mention coverage and sufficient context diversity. Unfortunately, when scaling NER to open situations, these advantages may no longer exist. And therefore it raises a critical question of whether previous creditable approaches can still work well when facing these challenges. As there is no currently available dataset to investigate this problem, this paper proposes to conduct randomization test on standard benchmarks. Specifically, we erase name regularity, mention coverage and context diversity respectively from the benchmarks, in order to explore their impact on the generalization ability of models. To further verify our conclusions, we also construct a new open NER dataset that focuses on entity types with weaker name regularity and lower mention coverage to verify our conclusion. From both randomization test and empirical experiments, we draw the conclusions that 1) name regularity is critical for the models to generalize to unseen mentions; 2) high mention coverage may undermine the model generalization ability and 3) context patterns may not require enormous data to capture when using pretrained encoders.

Motivation & Objective

• To investigate whether state-of-the-art NER models trained on standard benchmarks generalize effectively to open-world scenarios with weaker name regularity, lower mention coverage, and limited context diversity.
• To identify which factors—name regularity, mention coverage, or context diversity—most critically affect model generalization in open NER.
• To address the lack of benchmarks for open NER by proposing a systematic randomization test to isolate and evaluate the impact of specific data properties.
• To construct a new open NER dataset focusing on entity types with weak name regularity and low mention coverage to empirically validate findings.
• To provide actionable insights for building more robust and efficient NER models in low-resource and open-domain settings.

Proposed method

• Conducting randomization tests on standard NER benchmarks (e.g., CoNLL03, ACE2005, TAC-KBP) by systematically erasing name regularity, mention coverage, and context diversity.
• Applying mention replacement mechanisms to erase name regularity (e.g., permuting first and last names), mention coverage (e.g., replacing test mentions with unseen synonyms), and context diversity (e.g., reducing context length or replacing with random tokens).
• Training and evaluating the same fine-tuned BERT-based models on both original (vanilla) and randomized versions of the benchmarks to measure performance drops.
• Constructing a new open NER dataset from Wikipedia using entity types with weak name regularity (e.g., movie titles) and low mention coverage to validate findings.
• Analyzing model performance across varying training data sizes (500 to 10,000 instances) to assess the diminishing returns of data for context pattern learning.
• Using both quantitative performance drops and qualitative analysis to assess model reliance on specific data properties.

Experimental results

Research questions

• RQ1 To what extent does name regularity in training data affect a model’s ability to generalize to unseen mentions in open-world NER?
• RQ2 How does high mention coverage in standard benchmarks bias model generalization, and does it undermine performance on out-of-distribution mentions?
• RQ3 Is extensive training data necessary to learn context patterns when using pretrained encoders, or can models generalize effectively with minimal data?
• RQ4 Can the performance gap between in-dictionary and out-of-dictionary mentions be attributed to name regularity, and how significant is this effect?
• RQ5 To what extent do current benchmarks misrepresent the challenges of real-world open NER due to their inherent data properties?

Key findings

• Name regularity is critical for generalization: models lose over 24% F1 on precision and 18% on recall when name regularity is erased, indicating strong reliance on compositional patterns.
• High mention coverage undermines generalization: models trained on high-coverage data perform poorly on unseen mentions, suggesting overfitting to in-dictionary examples.
• Context patterns can be learned effectively with minimal data: performance improvements plateau after 3,000 training instances, indicating redundancy in large-scale data for context learning.
• The performance gap between in-dictionary and out-of-dictionary mentions exceeds 24% in F1 on the Wikipedia-based open NER dataset, confirming the importance of name regularity.
• Randomization tests reveal that models are most sensitive to name regularity, moderately affected by mention coverage, and least sensitive to context diversity when using pretrained encoders.
• The study confirms that standard benchmarks overestimate model generalization due to artificial data properties, necessitating new evaluation protocols for open NER.