[论文解读] LTP: A New Active Learning Strategy for Bert-CRF Based Named Entity Recognition

Mingyi Liu, Zhiying Tu+2 more|arXiv (Cornell University)|Jan 8, 2020

Topic Modeling参考文献 34被引用 21

一句话总结

本文提出 LTP（最低标记概率）这一基于不确定性的主动学习策略，用于 BERT-CRF 命名实体识别，通过结合 CRF 的输入与输出来选择信息量丰富的样本，而无需修改模型。LTP 通过偏好较短序列，在减少标注成本的同时，相较于传统方法，能以更少的标注标记实现更优性能。

ABSTRACT

In recent years, deep learning has achieved great success in many natural language processing tasks including named entity recognition. The shortcoming is that a large amount of manually-annotated data is usually required. Previous studies have demonstrated that active learning could elaborately reduce the cost of data annotation, but there is still plenty of room for improvement. In real applications we found existing uncertainty-based active learning strategies have two shortcomings. Firstly, these strategies prefer to choose long sequence explicitly or implicitly, which increase the annotation burden of annotators. Secondly, some strategies need to invade the model and modify to generate some additional information for sample selection, which will increase the workload of the developer and increase the training/prediction time of the model. In this paper, we first examine traditional active learning strategies in a specific case of BiLstm-CRF that has widely used in named entity recognition on several typical datasets. Then we propose an uncertainty-based active learning strategy called Lowest Token Probability (LTP) which combines the input and output of CRF to select informative instance. LTP is simple and powerful strategy that does not favor long sequences and does not need to invade the model. We test LTP on multiple datasets, and the experiments show that LTP performs slightly better than traditional strategies with obviously less annotation tokens on both sentence-level accuracy and entity-level F1-score. Related code have been release on this https URL

研究动机与目标

• 解决现有基于不确定性的主动学习策略在命名实体识别中的局限性。
• 通过避免在样本选择中偏好长序列，减轻标注负担。
• 开发一种无需模型修改或增加训练/推理时间的策略。
• 在极少标注数据下，提升句子级别准确率和实体级别 F1 分数。
• 为现实世界 NLP 应用提供轻量、可部署的解决方案。

提出的方法

• LTP 基于 CRF 输出序列中任意标记的最低预测概率来选择实例。
• 它结合 BERT 的输入表示与 CRF 的发射分数，以计算标记级别的不确定性。
• 该方法无需修改模型架构，也无需额外的推理计算。
• 样本选择基于序列中所有标记的最小概率，偏好不确定但简洁的实例。
• LTP 通过关注单个标记的置信度而非序列级别的不确定性，避免对长序列的偏好。
• 该策略无需修改模型，可轻松集成到现有的 BERT-CRF 流程中。

实验结果

研究问题

• RQ1 LTP 在标注效率方面与传统基于不确定性的主动学习策略相比如何？
• RQ2 LTP 是否通过避免对长序列的偏好，减轻了标注负担？
• RQ3 LTP 是否能在比基线方法更少的标注标记下实现更优性能？
• RQ4 LTP 在不修改模型的前提下，能在多大程度上保持或提升模型性能？
• RQ5 LTP 在多个命名实体识别数据集上的表现如何？

主要发现

• LTP 在句子级别准确率和实体级别 F1 分数上均略优于传统主动学习策略。
• LTP 显著减少了所需的标注标记数量，从而降低了标注成本。
• 该策略不偏好长序列，减轻了人工标注者的负担。
• LTP 无需模型修改或额外的训练/推理时间，因此高效且易于部署。
• 在多个数据集上的实验验证了 LTP 在不同 NER 设置下的有效性和鲁棒性。
• 作者发布了 LTP 的代码，支持可复现性与实际应用。