[论文解读] How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain

Mingchen Li, Rui Zhang|arXiv (Cornell University)|Jul 1, 2023

Topic Modeling被引用 6

一句话总结

本文研究了小语言模型（SLMs）和大语言模型（LLMs）在少样本医学命名实体识别（NER）中的表现，提出了一种检索与思维（Retrieval-and-Thinking, RT）框架，通过检索相关样例并应用逐步推理来提升LLMs的性能。RT在两个公开医学NER基准上显著优于强基线模型，取得了接近最先进水平的结果，但误识别和符号生成错误等挑战仍导致无法实现100%的准确率。

ABSTRACT

Recent advancements in language models (LMs) have led to the emergence of powerful models such as Small LMs (e.g., T5) and Large LMs (e.g., GPT-4). These models have demonstrated exceptional capabilities across a wide range of tasks, such as name entity recognition (NER) in the general domain. (We define SLMs as pre-trained models with fewer parameters compared to models like GPT-3/3.5/4, such as T5, BERT, and others.) Nevertheless, their efficacy in the medical section remains uncertain and the performance of medical NER always needs high accuracy because of the particularity of the field. This paper aims to provide a thorough investigation to compare the performance of LMs in medical few-shot NER and answer How far is LMs from 100\% Few-shot NER in Medical Domain, and moreover to explore an effective entity recognizer to help improve the NER performance. Based on our extensive experiments conducted on 16 NER models spanning from 2018 to 2023, our findings clearly indicate that LLMs outperform SLMs in few-shot medical NER tasks, given the presence of suitable examples and appropriate logical frameworks. Despite the overall superiority of LLMs in few-shot medical NER tasks, it is important to note that they still encounter some challenges, such as misidentification, wrong template prediction, etc. Building on previous findings, we introduce a simple and effective method called extsc{RT} (Retrieving and Thinking), which serves as retrievers, finding relevant examples, and as thinkers, employing a step-by-step reasoning process. Experimental results show that our proposed extsc{RT} framework significantly outperforms the strong open baselines on the two open medical benchmark datasets

研究动机与目标

• 比较SLMs和LLMs在少样本医学NER任务中的表现。
• 探究在通用领域训练的NER模型是否能有效迁移到医学领域。
• 分析标注数据量和质量对SLMs和LLMs在医学NER中表现的影响。
• 识别当前语言模型无法实现100%少样本医学NER准确率的剩余障碍。

提出的方法

• 提出一种检索与思维（RT）框架，作为少样本NER中的检索器与推理器。
• 利用语义相似度检索上下文相关样例，以提升提示质量。
• 应用逐步推理，引导LLMs通过逻辑化的实体抽取过程。
• 根据数据集特定性能，采用不同的样例选择策略——仅使用正样本或结合正负样本。
• 使用输出模板约束预测格式，减少结构错误。
• 在两个公开医学NER数据集（BC5CDR和NCBI）上，使用Micro-F1和精确匹配F1评估性能。

Figure 1: LLM performance on without (w/o) mask operation and with (w/) mask operation

实验结果

研究问题

• RQ1 在少样本医学NER任务中，SLMs与LLMs哪种模型类型表现更优？
• RQ2 在通用领域训练的NER模型能否成功迁移到医学领域？
• RQ3 标注样例的数量与质量如何影响SLMs和LLMs在医学NER中的表现？
• RQ4 哪些因素阻碍了当前最先进模型实现100%少样本医学NER准确率？

主要发现

• 当提供高质量指令和适当的逻辑框架时，LLMs在少样本医学NER中优于SLMs。
• 仅当模型在医学数据上进行特定预训练时，通用领域NER模型才能有效迁移到医学领域。
• 在低数据条件下，LLMs表现出比SLMs更稳定的性能，尤其是在训练数据部分标注的情况下。
• 检索样例的有效性因数据集而异；在NCBI数据集中，仅使用正样本效果更优，而在BC5CDR数据集中，结合正负样本效果更佳。
• RT在BC5CDR和NCBI两个数据集上均实现了优于或相当强基线模型的性能，尤其在1-shot和5-shot设置下表现突出。
• 尽管性能有所提升，RT仍因误识别、符号生成错误、错误模板预测以及部分标注数据中的类别冲突等问题，无法达到100%准确率。

Figure 2: The output format of different ICL methods