[Paper Review] A Practical Incremental Learning Framework For Sparse Entity Extraction

Hussein S. Al-Olimat, Steven Gustafson+3 more|arXiv (Cornell University)|Jun 26, 2018

Topic Modeling23 references4 citations

TL;DR

This paper proposes an incremental, interactive learning framework that combines Active Learning (AL) and Entity Set Expansion (ESE) to reduce annotation cost in sparse entity extraction. By using auto-annotation modes (FA, HFA, UFA) and an online evaluation method for flexible stopping criteria, the framework achieves 0.9–1.0 F-Score with 45–85% lower annotation cost across three datasets, demonstrating robust performance on low-resource entity classes.

ABSTRACT

This work addresses challenges arising from extracting entities from textual data, including the high cost of data annotation, model accuracy, selecting appropriate evaluation criteria, and the overall quality of annotation. We present a framework that integrates Entity Set Expansion (ESE) and Active Learning (AL) to reduce the annotation cost of sparse data and provide an online evaluation method as feedback. This incremental and interactive learning framework allows for rapid annotation and subsequent extraction of sparse data while maintaining high accuracy. We evaluate our framework on three publicly available datasets and show that it drastically reduces the cost of sparse entity annotation by an average of 85% and 45% to reach 0.9 and 1.0 F-Scores respectively. Moreover, the method exhibited robust performance across all datasets.

Motivation & Objective

• Address the high cost and low efficiency of annotating sparse entity classes in enterprise and domain-specific corpora.
• Overcome data sparsity and labeling starvation in supervised sequence labeling by integrating ESE and AL.
• Enable rapid, high-accuracy entity extraction through auto-annotation modes and flexible stopping criteria without requiring full gold-standard evaluation.
• Provide an online model confidence evaluation method to support incremental learning and reduce dependency on gold-standard annotations.
• Develop a usable, scalable framework for real-world deployment in resource-constrained settings.

Proposed method

• Employ pool-based Active Learning (AL) with a linear-chain Conditional Random Field (CRF) model to iteratively select the most informative sentences for annotation.
• Integrate Entity Set Expansion (ESE) to discover semantically similar entities by analogy, accelerating learning from few seed examples.
• Implement three auto-annotation modes—Fast (FA), Hyper Fast (HFA), and Ultra Fast (UFA)—to pre-annotate sentences and reduce manual labeling effort.
• Use an online evaluation method based on model confidence estimation (Equation 3 and 4) to determine stopping criteria without gold-standard data.
• Apply flexible stopping criteria that halt annotation when model confidence reaches a threshold, balancing cost and performance.
• Integrate the framework into Stanford CoreNLP for broad usability and open-source availability.

Experimental results

Research questions

• RQ1 Can combining ESE and AL reduce the annotation cost of sparse entity extraction while maintaining high model accuracy?
• RQ2 How effective are different auto-annotation modes (FA, HFA, UFA) in reducing the number of sentences requiring manual labeling?
• RQ3 To what extent can an online evaluation method based on model confidence replace gold-standard evaluation in stopping criteria?
• RQ4 How does the proposed framework compare to prior work, such as Tsuruoka et al. (2008), in terms of coverage and annotation efficiency?
• RQ5 Does the framework maintain robust performance across diverse, low-resource entity classes and datasets?

Key findings

• The framework reduced annotation cost by 45% to 85% across three public datasets while achieving 1.0 and 0.9 F-Score, respectively.
• The EAL query strategy outperformed Tsuruoka et al. (2008) by requiring 2,800–4,000 fewer sentences to reach the same coverage on overlapping entity classes.
• The HFA auto-annotation mode achieved a 33% reduction in annotation effort with only a 10% drop in F-Score, making it highly practical for enterprise use.
• The FA mode reduced manual annotation by 65% to reach 0.99 F-Score on the Location class, demonstrating strong cost-performance trade-offs.
• The online evaluation method showed a mean error margin of only 3% when estimating F-Score at 1.0, indicating high reliability in model confidence estimation.
• Despite some overestimation in early F-Score curves due to missing gold-standard annotations (e.g., in GENIA-Cell-Type and GENIA-Virus), the framework maintained robust performance across all datasets.