[Paper Review] Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI

Muhammad Aurangzeb Ahmad, Ilker Yaramis+1 more|arXiv (Cornell University)|Sep 26, 2023

Artificial Intelligence in Healthcare and Education14 citations

TL;DR

The paper discusses monitoring, measuring, and mitigating hallucinations in healthcare-focused LLMs to enable trustworthy, transparent, and responsible use, detailing evaluation methods, mitigation strategies, and future outlook.

ABSTRACT

Large language models have proliferated across multiple domains in as short period of time. There is however hesitation in the medical and healthcare domain towards their adoption because of issues like factuality, coherence, and hallucinations. Give the high stakes nature of healthcare, many researchers have even cautioned against its usage until these issues are resolved. The key to the implementation and deployment of LLMs in healthcare is to make these models trustworthy, transparent (as much possible) and explainable. In this paper we describe the key elements in creating reliable, trustworthy, and unbiased models as a necessary condition for their adoption in healthcare. Specifically we focus on the quantification, validation, and mitigation of hallucinations in the context in healthcare. Lastly, we discuss how the future of LLMs in healthcare may look like.

Motivation & Objective

• Motivate cautious but proactive adoption of LLMs in healthcare by addressing hallucinatory outputs.
• Define hallucinations in healthcare AI and identify their impact on safety, bias, privacy, and liability.
• Propose a framework for evaluating and mitigating hallucinations across data, models, and deployment contexts.
• Highlight practical guardrails and future directions for trustworthy healthcare LLMs.

Proposed method

• Discuss two evaluation scenarios: internal model access and black-box outputs, emphasizing self-check and interpretability.
• Describe human evaluation and automated metrics for factuality and alignment (e.g., FactScore, FActuality concepts) to gauge hallucinations.
• Outline a multi-pronged mitigation strategy including HITL, algorithmic corrections, fine-tuning, prompting improvements, adversarial training, input validation, memory augmentation, and model selection.
• Address benchmark reliability and the need for domain expert verification of datasets and outputs before inclusion in training or evaluation.
• Provide a forward-looking discussion on regulatory, safety, and practical adoption considerations in healthcare AI.

Experimental results

Research questions

• RQ1 How can hallucinations in healthcare LLMs be effectively evaluated and quantified when access to the model may be limited?
• RQ2 What combination of human-in-the-loop, algorithmic, and data-centered strategies best mitigates hallucinations in high-stakes healthcare tasks?
• RQ3 What benchmarks and evaluation practices ensure veracity and reliability of healthcare LLM outputs?
• RQ4 What is the role of model choice and external knowledge sources in reducing hallucinations in healthcare applications?

Key findings

• Hallucinations are a fundamental impediment to adopting LLMs in healthcare due to their potential to misinform on diagnosis or treatment.
• Evaluation can be performed via model-internal checks or black-box output analysis, supplemented by human and automated methods.
• Mitigation is multi-pronged, including HITL involvement, prompting strategies, fine-tuning, adversarial training, input validation, and external memory integration.
• Benchmark data must be verified by domain experts to avoid basing models on datasets containing false information.
• Tools and approaches like NeMo Guardrails are emerging to detect and manage hallucinations in healthcare contexts.