[论文解读] An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing

Sonish Sivarajkumar, Mark Kelley+3 more|arXiv (Cornell University)|Sep 14, 2023

Topic Modeling被引用 10

一句话总结

该论文通过在五个任务中使用 GPT-3.5、BARD 和 LLAMA2，实证评估零-shot 临床自然语言处理的提示策略，提出启发式提示和集成提示，并比较零-shot 与少-shot 提示。

ABSTRACT

Large language models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), especially in domains where labeled data is scarce or expensive, such as clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches. In this paper, we present a comprehensive and systematic experimental study on prompt engineering for five clinical NLP tasks: Clinical Sense Disambiguation, Biomedical Evidence Extraction, Coreference Resolution, Medication Status Extraction, and Medication Attribute Extraction. We assessed the prompts proposed in recent literature, including simple prefix, simple cloze, chain of thought, and anticipatory prompts, and introduced two new types of prompts, namely heuristic prompting and ensemble prompting. We evaluated the performance of these prompts on three state-of-the-art LLMs: GPT-3.5, BARD, and LLAMA2. We also contrasted zero-shot prompting with few-shot prompting, and provide novel insights and guidelines for prompt engineering for LLMs in clinical NLP. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative AI, and we hope that it will inspire and inform future research in this area.

研究动机与目标

• 研究在使用大型语言模型时，提示策略如何影响零-shot 零样本临床 NLP 的表现。
• 系统性比较来自近期文献和新提示的一系列提示类型。
• 为在临床 NLP 中使用大语言模型的提示工程提供实用指南。

提出的方法

• 在五个临床 NLP 任务上评估提示：Clinical Sense Disambiguation、Biomedical Evidence Extraction、Coreference Resolution、Medication Status Extraction 和 Medication Attribute Extraction。
• 测试来自文献的提示，如简单前缀、简单完形填空、思路链及预期性提示。
• 引入两种新提示类型：启发式提示和集成提示。
• 在三种最先进的语言模型 GPT-3.5、BARD 和 LLAMA2 上比较零-shot 提示与少-shot 提示。
• 分析提示方法的优点和缺点，以制定可操作的指南。

实验结果

研究问题

• RQ1 不同的提示策略如何在多任务和多模型下影响零-shot 临床 NLP 的表现？
• RQ2 启发式和集成提示是否相对于传统提示类型在临床 NLP 中带来改进？
• RQ3 在该领域中，零-shot 与少-shot 提示之间的权衡是什么？
• RQ4 在各种提示策略下，GPT-3.5、BARD 和 LLAMA2 在临床 NLP 任务中的表现如何比较？

主要发现

• 来自近期文献的提示在不同任务和模型上的效果各异。
• 提出并评估两种新颖的提示类型：启发式提示和集成提示。
• 对比零-shot 提示与少-shot 提示以确定在临床 NLP 中提示工程的实用指南。
• 本研究提供见解和指南，以促进未来在临床 NLP 中的提示工程研究。