[Paper Review] LTP: A New Active Learning Strategy for Bert-CRF Based Named Entity Recognition

Mingyi Liu, Zhiying Tu+2 more|arXiv (Cornell University)|Jan 8, 2020

Topic Modeling34 references21 citations

TL;DR

This paper proposes LTP (Lowest Token Probability), an uncertainty-based active learning strategy for BERT-CRF named entity recognition that selects informative samples by combining CRF input and output without modifying the model. LTP reduces annotation cost by favoring shorter sequences and achieving better performance with fewer labeled tokens compared to traditional methods.

ABSTRACT

In recent years, deep learning has achieved great success in many natural language processing tasks including named entity recognition. The shortcoming is that a large amount of manually-annotated data is usually required. Previous studies have demonstrated that active learning could elaborately reduce the cost of data annotation, but there is still plenty of room for improvement. In real applications we found existing uncertainty-based active learning strategies have two shortcomings. Firstly, these strategies prefer to choose long sequence explicitly or implicitly, which increase the annotation burden of annotators. Secondly, some strategies need to invade the model and modify to generate some additional information for sample selection, which will increase the workload of the developer and increase the training/prediction time of the model. In this paper, we first examine traditional active learning strategies in a specific case of BiLstm-CRF that has widely used in named entity recognition on several typical datasets. Then we propose an uncertainty-based active learning strategy called Lowest Token Probability (LTP) which combines the input and output of CRF to select informative instance. LTP is simple and powerful strategy that does not favor long sequences and does not need to invade the model. We test LTP on multiple datasets, and the experiments show that LTP performs slightly better than traditional strategies with obviously less annotation tokens on both sentence-level accuracy and entity-level F1-score. Related code have been release on this https URL

Motivation & Objective

• To address the limitations of existing uncertainty-based active learning strategies in named entity recognition.
• To reduce the annotation burden by avoiding preference for long sequences in sample selection.
• To develop a strategy that does not require model modification or increased training/prediction time.
• To improve performance in both sentence-level accuracy and entity-level F1-score with minimal labeled data.
• To provide a lightweight, deployable solution for real-world NLP applications.

Proposed method

• LTP selects instances based on the lowest predicted probability of any token in the CRF output sequence.
• It combines the input representation from BERT with the CRF's emission scores to compute token-level uncertainty.
• The method does not require model architecture changes or additional inference computation.
• Samples are selected based on the minimum probability across all tokens in a sequence, favoring uncertain but concise instances.
• LTP avoids bias toward long sequences by focusing on individual token confidence rather than sequence-level uncertainty.
• The strategy is implemented without modifying the model, making it easily integrable into existing BERT-CRF pipelines.

Experimental results

Research questions

• RQ1 How does LTP compare to traditional uncertainty-based active learning strategies in terms of annotation efficiency?
• RQ2 Does LTP reduce the annotation burden by avoiding preference for long sequences?
• RQ3 Can LTP achieve better performance with fewer labeled tokens compared to baseline methods?
• RQ4 To what extent does LTP maintain or improve model performance without model modification?
• RQ5 How does LTP perform across multiple named entity recognition datasets?

Key findings

• LTP achieves slightly better performance than traditional active learning strategies in both sentence-level accuracy and entity-level F1-score.
• LTP significantly reduces the number of annotation tokens required, leading to lower annotation cost.
• The strategy does not favor long sequences, reducing the burden on human annotators.
• LTP does not require model modification or additional training/prediction time, making it efficient and deployable.
• Experiments on multiple datasets confirm LTP's effectiveness and robustness across different NER settings.
• The authors released code for LTP, enabling reproducibility and practical adoption.