[Paper Review] Generalisation in Named Entity Recognition: A Quantitative Analysis

Isabelle Augenstein, Leon Derczynski+1 more|White Rose Research Online (University of Leeds, The University of Sheffield, University of York)|Jan 11, 2017

Topic Modeling43 references4 citations

TL;DR

This paper quantitatively analyzes generalization challenges in Named Entity Recognition (NER) across diverse text genres, finding that unseen named entities and out-of-vocabulary (OOV) features significantly reduce F1 scores—especially in social media and web content. It demonstrates that state-of-the-art NER models, particularly those relying on surface form memorization, struggle to generalize beyond training data, with SENNA showing the best performance due to word embeddings, while performance correlates most strongly with the proportion of unseen named entities rather than corpus size or feature sparsity.

ABSTRACT

Named Entity Recognition (NER) is a key NLP task, which is all the more challenging on Web and user-generated content with their diverse and continuously changing language. This paper aims to quantify how this diversity impacts state-of-the-art NER methods, by measuring named entity (NE) and context variability, feature sparsity, and their effects on precision and recall. In particular, our findings indicate that NER approaches struggle to generalise in diverse genres with limited training data. Unseen NEs, in particular, play an important role, which have a higher incidence in diverse genres such as social media than in more regular genres such as newswire. Coupled with a higher incidence of unseen features more generally and the lack of large training corpora, this leads to significantly lower F1 scores for diverse genres as compared to more regular ones. We also find that leading systems rely heavily on surface forms found in training data, having problems generalising beyond these, and offer explanations for this observation.

Motivation & Objective

• Many NER systems fail to generalize to diverse, user-generated content like social media due to linguistic variation and unseen entities.
• Existing studies lack quantitative analysis of how entity diversity, context variability, and OOV features impact NER performance across genres.
• The study aims to identify the primary drivers of performance drop in out-of-domain NER, especially in low-resource, diverse corpora.
• Researchers seek to evaluate whether memorization of training data surface forms explains poor generalization, and whether larger or domain-adapted training data can mitigate this.
• Understanding the role of unseen named entities and features is critical for improving robustness in real-world NLP applications.

Proposed method

• Experiments compare three state-of-the-art NER systems—Stanford NER, SENNA, and CRFSuite—across six benchmark corpora spanning newswire, broadcast news, weblogs, telephone speech, and Usenet.
• Corpora include diverse genres such as CoNLL (newswire), MUC-7 (newswire), ACE (multi-genre), OntoNotes (multi-genre), Ritter Twitter, and MSM 2013, ensuring broad coverage.
• Performance is evaluated using F1 score, precision, and recall, with a focus on unseen named entities and OOV features.
• Unseen named entities are defined as those appearing in test sets but not in training sets; OOV features are lexical or syntactic features not seen during training.
• A memorization baseline is used to predict performance by selecting the most frequent NE label for each token sequence observed in training.
• Corpus analysis quantifies entity diversity, context variability, and feature sparsity to correlate these with system performance.

Experimental results

Research questions

• RQ1 RQ1: How does NERC performance differ across various NER approaches on diverse corpora?
• RQ2 RQ2: How does NERC performance vary across different text genres and corpora?
• RQ3 RQ3: What is the impact of named entity diversity on system performance?
• RQ4 RQ4: What is the relationship between OOV features, OOV entities, and F1 score?
• RQ5 RQ5: How well do NER methods generalize out-of-domain, and what is the impact of unseen named entities on performance?

Key findings

• SENNA achieves the highest F1 scores across all corpora, outperforming Stanford NER and CRFSuite, largely due to its use of deep learning-based word embeddings.
• Despite being the largest corpus in terms of NE count, OntoNotes NW yields lower F1 than CoNLL, indicating that corpus size alone does not ensure better performance.
• Corpora with higher proportions of unseen named entities show significantly lower F1, with performance on unseen NEs being about 17 points lower than on seen NEs across all models.
• CoNLL newswire corpus achieves the highest F1 because it has the lowest proportion of unseen named entities, confirming that entity diversity is a key predictor of performance.
• Memorization baseline—predicting the most frequent NE label per token sequence—strongly predicts system accuracy, indicating that models rely heavily on memorizing training data surface forms.
• Out-of-domain training leads to significant performance drops unless the out-of-domain memorization baseline outperforms the in-domain one, suggesting that domain alignment is critical for generalization.