[Paper Review] Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information Access

Ziang Xiao, Q. Vera Liao+3 more|arXiv (Cornell University)|Jan 25, 2023

Misinformation and Its Impacts4 citations

TL;DR

This paper presents an expert-sourcing framework to develop Jennifer, an AI chatbot that delivers credible, up-to-date, and easily accessible COVID-19 health information by leveraging a global network of 150+ scientists and health professionals. The chatbot effectively supports information seekers by improving retrieval accuracy, user satisfaction, and trust, while the framework enables rapid content updates and high-quality curation through decentralized expert collaboration and a two-stage verification process.

ABSTRACT

During a public health crisis like the COVID-19 pandemic, a credible and easy-to-access information portal is highly desirable. It helps with disease prevention, public health planning, and misinformation mitigation. However, creating such an information portal is challenging because 1) domain expertise is required to identify and curate credible and intelligible content, 2) the information needs to be updated promptly in response to the fast-changing environment, and 3) the information should be easily accessible by the general public; which is particularly difficult when most people do not have the domain expertise about the crisis. In this paper, we presented an expert-sourcing framework and created Jennifer, an AI chatbot, which serves as a credible and easy-to-access information portal for individuals during the COVID-19 pandemic. Jennifer was created by a team of over 150 scientists and health professionals around the world, deployed in the real world and answered thousands of user questions about COVID-19. We evaluated Jennifer from two key stakeholders' perspectives, expert volunteers and information seekers. We first interviewed experts who contributed to the collaborative creation of Jennifer to learn about the challenges in the process and opportunities for future improvement. We then conducted an online experiment that examined Jennifer's effectiveness in supporting information seekers in locating COVID-19 information and gaining their trust. We share the key lessons learned and discuss design implications for building expert-sourced and AI-powered information portals, along with the risks and opportunities of misinformation mitigation and beyond.

Motivation & Objective

• To address the challenge of providing timely, accurate, and understandable health information during public health crises like COVID-19.
• To reduce misinformation and information overload by creating a centralized, expert-curated information portal accessible to the general public.
• To evaluate the effectiveness of an expert-sourced AI chatbot from both expert volunteers’ and information seekers’ perspectives.
• To identify design implications for scalable, trustworthy, and responsive AI-powered health information systems.
• To explore how large language models can be integrated with expert-curated knowledge to enhance engagement without compromising factual accuracy.

Proposed method

• Developed a decentralized expert-sourcing framework with a hierarchical structure and two-stage verification to ensure content quality and scalability.
• Assembled a global team of 150+ health and science experts to curate, verify, and update content in real time during the pandemic.
• Built Jennifer, an AI chatbot that answers user queries using a knowledge base of expert-verified, plain-language responses.
• Conducted semi-structured interviews with expert volunteers to identify challenges and needs in content creation and collaboration.
• Performed an online experiment with information seekers to assess Jennifer’s effectiveness in information retrieval, trust, and satisfaction.
• Integrated feedback loops using daily conversation logs to iteratively improve the chatbot’s performance and responsiveness.

Figure 1 . The figures show the architecture overview of Jennifer . Our expert sourcing framework extends Juji’s base system’s ability to answer people’s questions about COVID-19. Through a hierarchical structure, expert volunteers were divided into four groups, Admins, Curator, Helper, and Tester,

Experimental results

Research questions

• RQ1 How can an expert-sourcing framework be designed to support the rapid, accurate, and credible curation of health information during a fast-evolving crisis?
• RQ2 What challenges do expert volunteers face in contributing to a large-scale, real-time AI chatbot, and what technological support is needed?
• RQ3 To what extent can an expert-sourced AI chatbot improve information retrieval, user satisfaction, and trust among non-expert users?
• RQ4 How can proactive design features, such as notifications, support repeated information-seeking behaviors common during prolonged crises?
• RQ5 What role can large language models play in enhancing engagement while preserving factual accuracy in high-stakes health information contexts?

Key findings

• Jennifer successfully supported information seekers in retrieving accurate and relevant information, with high user satisfaction and trust in the chatbot’s responses.
• Expert volunteers reported that the lack of a centralized tracking system and real-time collaboration tools were major challenges in content updating and coordination.
• The two-stage verification process and hierarchical team structure enabled scalable collaboration among diverse experts while maintaining information quality.
• Information seekers frequently returned to Jennifer for updated information, indicating repeated and ongoing information needs during prolonged uncertainty.
• The framework demonstrated feasibility for real-world deployment, with thousands of user interactions and continuous content updates over six months.
• Proactive notification features could significantly reduce user effort in monitoring evolving information, suggesting a valuable extension for future systems.

(a) Jennifer opens the conversation with a self-introduction.