[Paper Review] ANEA: Distant Supervision for Low-Resource Named Entity Recognition

Michael A. Hedderich, Lukas Lange+1 more|arXiv (Cornell University)|Feb 25, 2021

Topic Modeling33 references4 citations

TL;DR

ANEA is an open-source tool that automates distant supervision for low-resource Named Entity Recognition (NER) by extracting entity names from Wikidata and enabling linguists to tune the annotation process with minimal manual effort. In six low-resource scenarios, ANEA boosted F1 scores by an average of 18 points when used to train neural NER models, significantly improving performance with under 30 minutes of expert interaction.

ABSTRACT

Distant supervision allows obtaining labeled training corpora for low-resource settings where only limited hand-annotated data exists. However, to be used effectively, the distant supervision must be easy to gather. In this work, we present ANEA, a tool to automatically annotate named entities in texts based on entity lists. It spans the whole pipeline from obtaining the lists to analyzing the errors of the distant supervision. A tuning step allows the user to improve the automatic annotation with their linguistic insights without labelling or checking all tokens manually. In six low-resource scenarios, we show that the F1-score can be increased by on average 18 points through distantly supervised data obtained by ANEA.

Motivation & Objective

• To address the lack of labeled training data in low-resource languages and domains for Named Entity Recognition (NER).
• To reduce manual annotation costs and time by automating the creation of distantly supervised training data.
• To enable language experts to improve automatic entity annotation using linguistic insights without manually labeling every token.
• To provide a scalable, user-friendly tool that supports both technical and non-technical users through a library and GUI.

Proposed method

• Automatically extracts entity names from Wikidata dumps using predefined categories (e.g., person, location) for target languages.
• Applies rule-based matching to annotate named entities in unlabeled text based on the extracted entity lists.
• Supports linguistic tuning via configurable preprocessing steps (e.g., lemmatization, case normalization) to balance precision and recall.
• Provides a graphical user interface and a Python library for flexible integration and deployment.
• Enables users to iteratively refine the annotation process by adjusting matching rules and evaluating impact on a validation set.
• Uses a back-end server with low memory footprint, supporting local or remote execution and compatibility with multiple NLP tools like SpaCy and EstNLTK.

Experimental results

Research questions

• RQ1 Can distant supervision via Wikidata-based entity lists effectively generate training data for low-resource NER in diverse languages and domains?
• RQ2 How can linguistic expertise be efficiently integrated into automatic entity annotation without requiring full manual labeling?
• RQ3 To what extent does ANEA’s tuning capability improve the quality of distantly supervised NER data compared to baseline methods?
• RQ4 How much performance gain does ANEA’s distantly supervised data provide when used to train neural NER models in low-resource settings?
• RQ5 Can ANEA be deployed efficiently across different technical backgrounds and system configurations?

Key findings

• In six low-resource NER scenarios, ANEA achieved the highest F1-score in all but one case, outperforming two baselines across all metrics.
• On average, ANEA increased the F1-score of downstream neural NER models by 18 points when using distantly supervised training data.
• The tuning step improved performance significantly, with gains of up to 88 F1 points in the En CONTI. (continent) setting when using tuned ANEA data.
• For 14 out of 16 evaluated entity types, ANEA’s distant supervision provided the largest performance improvement in Experiment B.
• The tool reduced manual effort to less than 30 minutes of expert interaction while achieving substantial performance gains, demonstrating high efficiency and practicality.