[Paper Review] Few-NERD: A Few-Shot Named Entity Recognition Dataset

Ning Ding, Guangwei Xu+6 more|arXiv (Cornell University)|May 16, 2021

Topic Modeling33 references5 citations

TL;DR

This paper introduces FEW-NERD, the first large-scale, human-annotated few-shot named entity recognition dataset with 8 coarse-grained and 66 fine-grained entity types across 188,238 Wikipedia sentences. It enables benchmarking of few-shot NER models through three distinct tasks—supervised, intra-type, and inter-type generalization—demonstrating that current models still struggle with fine-grained entity recognition, especially in low-shot settings.

ABSTRACT

Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practical and challenging task. Current approaches collect existing supervised NER datasets and re-organize them to the few-shot setting for empirical study. These strategies conventionally aim to recognize coarse-grained entity types with few examples, while in practice, most unseen entity types are fine-grained. In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238 sentences from Wikipedia, 4,601,160 words are included and each is annotated as context or a part of a two-level entity type. To the best of our knowledge, this is the first few-shot NER dataset and the largest human-crafted NER dataset. We construct benchmark tasks with different emphases to comprehensively assess the generalization capability of models. Extensive empirical results and analysis show that Few-NERD is challenging and the problem requires further research. We make Few-NERD public at https://ningding97.github.io/fewnerd/.

Motivation & Objective

• To address the lack of a standardized, large-scale benchmark for few-shot named entity recognition (NER) with fine-grained entity types.
• To overcome the limitations of existing datasets, which typically offer only 4–18 coarse-grained types and are repurposed for few-shot evaluation.
• To provide a unified benchmark that supports comprehensive evaluation of generalization capabilities in few-shot NER, including instance-level, type-level, and knowledge transfer generalization.
• To facilitate future research in few-shot NER and continual knowledge graph construction by offering a high-quality, publicly available dataset.

Proposed method

• The dataset is constructed from 188,238 Wikipedia sentences, manually annotated by trained annotators, resulting in 4.6 million words and 491,700 entities.
• A hierarchical annotation schema with 8 coarse-grained and 66 fine-grained entity types was designed through multiple pre-annotation rounds to ensure granularity and consistency.
• Three benchmark tasks were defined: FEW-NERD (SUP) for supervised learning, FEW-NERD (INTRA) for intra-type generalization, and FEW-NERD (INTER) for inter-type generalization.
• State-of-the-art few-shot NER models—ProtoBERT, NNShot, and StructShot—were evaluated on the three tasks to assess performance across different few-shot settings.
• Error analysis was conducted using token-level and type-level error breakdowns (e.g., FP, FN, Within, Outer) across five experimental runs to assess model stability and failure modes.
• The dataset is publicly released at https://ningding97.github.io/fewnerd/ along with model checkpoints to reduce carbon footprint and support reproducibility.

Experimental results

Research questions

• RQ1 How well do current few-shot NER models generalize to unseen, fine-grained entity types in low-shot settings?
• RQ2 To what extent can models transfer knowledge across different coarse-grained entity types in few-shot NER?
• RQ3 How do different few-shot learning strategies (e.g., prototypical networks, nearest neighbor, structured decoding) perform on a large-scale, fine-grained NER benchmark?
• RQ4 What are the main failure modes of current few-shot NER models, particularly in terms of span detection and type classification?
• RQ5 Can the proposed benchmark tasks (SUP, INTRA, INTER) effectively differentiate the strengths and weaknesses of few-shot NER models?

Key findings

• FEW-NERD is the first large-scale, human-annotated few-shot NER dataset with 66 fine-grained entity types, significantly expanding the scope of existing benchmarks.
• In the 5-way 5–10 shot setting, ProtoBERT achieved the best performance among the evaluated models, outperforming NNShot and StructShot, especially in type classification.
• StructShot showed improved precision in the FEW-NERD (INTRA) task, indicating that Viterbi decoding helps reduce false positives when knowledge transfer is difficult.
• Error analysis revealed that token-level detection is relatively robust (low FP and FN), but type misclassification—especially across coarse-grained categories (Outer errors)—remains a major challenge.
• The models exhibited instability in the nearest neighbor mechanism (NNShot and StructShot), while prototypical models showed greater stability due to the regularization effect of prototype computation.
• The dataset enables effective evaluation of generalization across instance, type, and knowledge transfer dimensions, highlighting the persistent difficulty of few-shot NER, especially for fine-grained entities.