[Paper Review] CataractBot: An LLM-Powered Expert-in-the-Loop Chatbot for Cataract Patients

Pragnya Ramjee, Bhuvan Sachdeva+5 more|arXiv (Cornell University)|Feb 7, 2024

Transportation and Mobility Innovations4 citations

TL;DR

CataractBot is a large language model (LLM)-powered, expert-in-the-loop chatbot that delivers real-time, accurate cataract surgery information via WhatsApp by retrieving from a curated knowledge base and asynchronously verifying responses with medical experts. In a real-world deployment with 49 patients and attendants, it improved information access, reduced anxiety, and built trust through expert validation, demonstrating feasibility for scalable, reliable patient support in low-resource healthcare settings.

ABSTRACT

The healthcare landscape is evolving, with patients seeking reliable information about their health conditions and available treatment options. Despite the abundance of information sources, the digital age overwhelms individuals with excess, often inaccurate information. Patients primarily trust medical professionals, highlighting the need for expert-endorsed health information. However, increased patient loads on experts has led to reduced communication time, impacting information sharing. To address this gap, we developed CataractBot. CataractBot answers cataract surgery related questions instantly using an LLM to query a curated knowledge base, and provides expert-verified responses asynchronously. It has multimodal and multilingual capabilities. In an in-the-wild deployment study with 49 patients and attendants, 4 doctors, and 2 patient coordinators, CataractBot demonstrated potential, providing anytime accessibility, saving time, accommodating diverse literacy levels, alleviating power differences, and adding a privacy layer between patients and doctors. Users reported that their trust in the system was established through expert verification. Broadly, our results could inform future work on expert-mediated LLM bots.

Motivation & Objective

• To address the growing patient demand for reliable, timely medical information about cataract surgery, especially amid rising doctor-patient ratios and limited consultation time.
• To reduce patient anxiety and improve health outcomes by providing accessible, accurate, and expert-verified information through a scalable digital solution.
• To design a system that combines the speed of LLMs with the reliability of expert verification, ensuring trustworthy responses without overwhelming clinical staff.
• To evaluate the feasibility and impact of an expert-in-the-loop chatbot in a real-world, low-literacy, resource-constrained healthcare environment like a tertiary eye hospital in India.
• To establish a framework for deploying similar expert-mediated LLM chatbots in other medical and non-medical domains, such as oncology or agriculture.

Proposed method

• The system uses Retrieval-Augmented Generation (RAG) over a custom, curated knowledge base of cataract-related information to generate real-time responses to patient queries.
• LLM-generated responses are asynchronously verified by medical experts (doctors and coordinators) who review and approve answers before they are shared with patients.
• The chatbot is deployed via WhatsApp, a widely used messaging platform, to ensure accessibility for patients with low digital literacy and limited smartphone experience.
• A knowledge base update mechanism involves a KB Update expert who reviews and integrates new medical content, with plans to scale to multiple contributors to reduce single-point failure.
• The system supports multilingual and multimodal interactions, accommodating diverse literacy levels and enabling image or text-based queries.
• Expert verification is embedded into the workflow, allowing doctors to verify responses during breaks or outside clinic hours, minimizing disruption to clinical routines.

Figure 1. Asking question to CataractBot , and getting a verified answer from the expert

Experimental results

Research questions

• RQ1 Can an LLM-powered chatbot with expert-in-the-loop verification effectively meet patients’ information needs for cataract surgery in a real-world, low-resource clinical setting?
• RQ2 How do patients and healthcare providers perceive trust and usability of a chatbot that combines LLM-generated responses with expert validation?
• RQ3 To what extent can such a system reduce patient anxiety and improve information access without increasing the workload of already overburdened medical staff?
• RQ4 What are the key technical and operational challenges in scaling an expert-verified LLM chatbot across a large patient population in a developing healthcare context?
• RQ5 How can the system maintain knowledge base accuracy and consistency when updates are made by multiple experts over time?

Key findings

• In a real-world deployment with 49 patients and attendants, CataractBot was perceived as valuable, with users appreciating its anytime accessibility and ability to provide answers outside clinic hours.
• Patients expressed high trust in the system, particularly because responses were verified by doctors, and were willing to wait for expert confirmation, understanding the doctors’ time constraints.
• The chatbot successfully accommodated users with varying literacy levels through multilingual and multimodal support, enhancing inclusivity in digital health interventions.
• Despite its success, the knowledge base became outdated during the study, and errors were introduced by the KB Update expert, highlighting the need for multiple contributors and automated update mechanisms.
• The system demonstrated that expert-in-the-loop verification is feasible in practice, with doctors and coordinators contributing verification during breaks and outside clinic hours, suggesting low operational friction.
• The study indicates that similar frameworks could be extended to other complex treatments like glaucoma surgery or cancer therapy, where information needs are higher and more nuanced.

Figure 2. Asking question to CataractBot through audio