[论文解读] Natural Language Programming in Medicine: Administering Evidence Based Clinical Workflows with Autonomous Agents Powered by Generative Large Language Models

Akhil Vaid, Joshua Lampert+28 more|arXiv (Cornell University)|Jan 5, 2024

Artificial Intelligence in Healthcare and Education被引用 7

一句话总结

该论文评估由生成型大语言模型驱动的自治代理，在模拟的三级护理环境中执行循证临床工作流程，比较专有模型和开源模型与RAG，并强调需要人为监督和基于NLP的行为修改。

ABSTRACT

Generative Large Language Models (LLMs) hold significant promise in healthcare, demonstrating capabilities such as passing medical licensing exams and providing clinical knowledge. However, their current use as information retrieval tools is limited by challenges like data staleness, resource demands, and occasional generation of incorrect information. This study assessed the potential of LLMs to function as autonomous agents in a simulated tertiary care medical center, using real-world clinical cases across multiple specialties. Both proprietary and open-source LLMs were evaluated, with Retrieval Augmented Generation (RAG) enhancing contextual relevance. Proprietary models, particularly GPT-4, generally outperformed open-source models, showing improved guideline adherence and more accurate responses with RAG. The manual evaluation by expert clinicians was crucial in validating models' outputs, underscoring the importance of human oversight in LLM operation. Further, the study emphasizes Natural Language Programming (NLP) as the appropriate paradigm for modifying model behavior, allowing for precise adjustments through tailored prompts and real-world interactions. This approach highlights the potential of LLMs to significantly enhance and supplement clinical decision-making, while also emphasizing the value of continuous expert involvement and the flexibility of NLP to ensure their reliability and effectiveness in healthcare settings.

研究动机与目标

• 在医学领域推动并评估使用自治LLM代理执行循证临床工作流程。
• 在三级护理仿真中比较专有与开源LLMs在指南遵循和回应准确性方面的表现。
• 评估检索增强生成（RAG）对上下文相关性和决策质量的影响。
• 展示自然语言编程作为一种在临床环境中安全调整模型行为的实用范式。

提出的方法

• 使用现实世界临床病例在多专业领域模拟三级护理医疗中心。
• 评估专有和开源LLMs在自治临床任务执行中的表现。
• 结合检索增强生成（RAG），以提升输出的上下文相关性。
• 应用专家临床医生的人工评估以验证模型输出。
• 倡导自然语言编程（NLP）作为通过提示和真实世界互动调整模型行为的范式。

实验结果

研究问题

• RQ1 自治LLM代理是否能够在模拟医院环境中在多专业领域可靠地遵循临床指南？
• RQ2 在使用RAG时，专有模型（如GPT-4）是否在指南遵循和准确性方面优于开源模型？
• RQ3 检索增强生成是否提高了LLM驱动的临床工作流程的上下文相关性和正确性？
• RQ4 在人类专家监督在验证和监控自治医疗代理中的作用是什么？
• RQ5 自然语言编程是否是一种可行且有效的方法，用于调整自治临床代理以实现可靠性和安全性？

主要发现

• 专有模型，尤其是GPT-4，在使用RAG时在指南遵循和准确性方面通常优于开源模型。
• RAG提升了医疗自治情境下回答的上下文相关性。
• 手动的专家临床医生评估对于验证模型输出和确保安全运行至关重要。
• NL 编程通过定制提示和互动实现对模型行为的精确调整。
• 该方法展示了LLMs在增强临床决策中的潜力，同时需要持续的专家参与。