[Paper Review] PharmacyGPT: The AI Pharmacist

Zhengliang Liu, Zihao Wu+13 more|arXiv (Cornell University)|Jul 19, 2023

Artificial Intelligence in Healthcare and Education11 citations

TL;DR

PharmacyGPT investigates ChatGPT and GPT-4 for pharmacy tasks, using dynamic prompting and iterative optimization to generate patient clusters, mortality and APACHE II predictions, and medication plans from ICU data. Results show GPT-4 with dynamic context performs best among tested approaches, with notable limitations.

ABSTRACT

In this study, we introduce PharmacyGPT, a novel framework to assess the capabilities of large language models (LLMs) such as ChatGPT and GPT-4 in emulating the role of clinical pharmacists. Our methodology encompasses the utilization of LLMs to generate comprehensible patient clusters, formulate medication plans, and forecast patient outcomes. We conduct our investigation using real data acquired from the intensive care unit (ICU) at the University of North Carolina Chapel Hill (UNC) Hospital. Our analysis offers valuable insights into the potential applications and limitations of LLMs in the field of clinical pharmacy, with implications for both patient care and the development of future AI-driven healthcare solutions. By evaluating the performance of PharmacyGPT, we aim to contribute to the ongoing discourse surrounding the integration of artificial intelligence in healthcare settings, ultimately promoting the responsible and efficacious use of such technologies.

Motivation & Objective

• Explore the capabilities of LLMs (ChatGPT and GPT-4) in pharmacy-related tasks.
• Develop dynamic prompting and iterative optimization to adapt LLMs to clinical pharmacy without fine-tuning.
• Generate interpretable patient clusters using LLM embeddings and clustering.
• Assess LLMs for predicting patient outcomes and generating medication plans in ICU data.

Proposed method

• Create interpretable patient clusters by generating 1536-dimensional GPT-3 embeddings from patient data and applying hierarchical clustering.
• Employ an iterative optimization algorithm that updates prompts based on evaluation scores from model outputs.
• Use a dynamic context approach to prompt construction to improve GPT-4/ChatGPT performance.
• Evaluate mortality and APACHE II score prediction performance via various few-shot prompting strategies (rand_5-shot, freq_5-shot, bcat_rand_5-shot, sim_5-shot, and GPT-4 variants).
• Compare generated medication plans from GPT-4 with expert review to discuss practicality and evaluation needs beyond ROUGE-like metrics.

Figure 1 : Interpretable Clusters for real ICU data

Experimental results

Research questions

• RQ1 Can LLMs generate interpretable patient clusters from ICU data using embeddings and clustering?
• RQ2 How well do ChatGPT and GPT-4 predict hospital mortality and APACHE II scores with dynamic prompts and few-shot demonstrations?
• RQ3 What is the potential and limitation of GPT-4/ChatGPT in generating ICU medication plans, and how should they be evaluated?
• RQ4 How can iterative prompt optimization improve LLM performance without model fine-tuning?

Key findings

• Clustering yielded groups aligning with ICD-10 categories and expert-validated interpretability.
• GPT-4 with dynamic context and similar samples achieved the highest accuracy among tested models for outcome prediction.
• Mortality prediction precision and F1 were affected by data imbalance (alive vs dead 9:1) and small deceased sample size (46 in test set).
• APACHE II score prediction saw GPT-4–based prompts outperform other approaches, indicating first-day data strong relevance.
• GPT-4 generated medication plans require expert pharmacist review and new, task-specific evaluation metrics beyond ROUGE.
• Imbalance and temporal changes in patient condition limit mortality predictability and evaluation from static initial-day data.

Figure 2 : Uncovering underlying neurological disorders of a group of patients