[Paper Review] Named Entity Recognition with Extremely Limited Data

John F. Foley, Sheikh Muhammad Sarwar+1 more|arXiv (Cornell University)|Jun 12, 2018

Topic Modeling22 references11 citations

TL;DR

This paper proposes Named Entity Search (NES), treating named entity recognition as a query-time retrieval task to address extremely limited labeled data. By modeling tokens as documents and using handcrafted features as query terms, the approach achieves high effectiveness with minimal training data, enabling interactive, efficient entity detection for rare or niche entity classes using standard search engines.

ABSTRACT

Traditional information retrieval treats named entity recognition as a pre-indexing corpus annotation task, allowing entity tags to be indexed and used during search. Named entity taggers themselves are typically trained on thousands or tens of thousands of examples labeled by humans. However, there is a long tail of named entities classes, and for these cases, labeled data may be impossible to find or justify financially. We propose exploring named entity recognition as a search task, where the named entity class of interest is a query, and entities of that class are the relevant "documents". What should that query look like? Can we even perform NER-style labeling with tens of labels? This study presents an exploration of CRF-based NER models with handcrafted features and of how we might transform them into search queries.

Motivation & Objective

• Address the challenge of named entity recognition (NER) for rare or niche entity classes with extremely limited labeled data.
• Overcome the limitations of traditional NER models that require thousands of annotated examples, which are often unfeasible for long-tail entity types.
• Explore an interactive, query-time approach to NER that avoids pre-training classifiers for each new entity class.
• Enable expert users to dynamically detect entities like 'Mediterranean islands' or 'cigarette brands' without prior model training.
• Demonstrate feasibility and effectiveness of using search engines to rank tokens based on entity class queries, even with only tens of labeled examples.

Proposed method

• Treat each token in a corpus as a document and represent its features (e.g., part-of-speech, word shape, n-grams) as terms in a search index.
• Construct queries for entity classes by combining handcrafted features such as word shape, part-of-speech tags, and Brown cluster embeddings.
• Use a CRF-based NER model as a baseline and translate its feature set into a retrieval model where feature weights inform query term weights.
• Apply standard information retrieval techniques (e.g., BM25) to score tokens based on their relevance to a given entity class query.
• Limit model size by selecting only the most informative features (e.g., top 1000) to improve efficiency without significant performance loss.
• Use active learning-like interaction: users refine queries iteratively based on ranked results, enabling exploration of sparse entity classes.

Experimental results

Research questions

• RQ1 How can CRF-based NER models with handcrafted features be effectively transformed into a retrieval model where tokens are documents?
• RQ2 Can a reasonable ranking of tokens be produced for user interaction when only a handful of labeled examples are available?
• RQ3 Can NER models be made small enough to be efficiently executed by a search engine without sacrificing effectiveness?
• RQ4 Which handcrafted NER features are most effective in this ad-hoc, retrieval-based NER setting?

Key findings

• The approach achieves nearly identical performance with models limited to 1000 features compared to full models, demonstrating that model compression does not significantly degrade effectiveness.
• Query execution time remains under one second for up to 500 query terms, and under 10 minutes even for 5000 terms, making the system practical for interactive use.
• The top-performing features for the QA-List dataset were 'words to the right', 'words to the left', 'char n-grams', 'word shape', and 'Brown cluster embeddings'.
• On the QA-List dataset, the interactive and unsure baselines showed comparable effectiveness, validating the approach for rare and complex entity classes.
• The learning curves for down-sampled models were nearly indistinguishable from full models, indicating that only a few hundred interactions are needed to achieve high performance.
• The system can process the full AQUAINT corpus in just a few CPU-minutes using a few hundred key features, compared to hours for traditional CRF training.