[论文解读] Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information Access

Ziang Xiao, Q. Vera Liao+3 more|arXiv (Cornell University)|Jan 25, 2023

Misinformation and Its Impacts被引用 4

一句话总结

本文提出了一种专家供稿框架，用于开发Jennifer这一AI聊天机器人，该机器人通过利用全球150多位科学家和卫生专业人员的网络，提供可信、及时且易于获取的COVID-19健康信息。该聊天机器人通过提升检索准确性、用户满意度和信任度，有效支持了信息获取者；同时，该框架通过去中心化的专家协作与两阶段验证流程，实现了内容的快速更新和高质量编纂。

ABSTRACT

During a public health crisis like the COVID-19 pandemic, a credible and easy-to-access information portal is highly desirable. It helps with disease prevention, public health planning, and misinformation mitigation. However, creating such an information portal is challenging because 1) domain expertise is required to identify and curate credible and intelligible content, 2) the information needs to be updated promptly in response to the fast-changing environment, and 3) the information should be easily accessible by the general public; which is particularly difficult when most people do not have the domain expertise about the crisis. In this paper, we presented an expert-sourcing framework and created Jennifer, an AI chatbot, which serves as a credible and easy-to-access information portal for individuals during the COVID-19 pandemic. Jennifer was created by a team of over 150 scientists and health professionals around the world, deployed in the real world and answered thousands of user questions about COVID-19. We evaluated Jennifer from two key stakeholders' perspectives, expert volunteers and information seekers. We first interviewed experts who contributed to the collaborative creation of Jennifer to learn about the challenges in the process and opportunities for future improvement. We then conducted an online experiment that examined Jennifer's effectiveness in supporting information seekers in locating COVID-19 information and gaining their trust. We share the key lessons learned and discuss design implications for building expert-sourced and AI-powered information portals, along with the risks and opportunities of misinformation mitigation and beyond.

研究动机与目标

• 为应对在COVID-19等公共卫生危机期间提供及时、准确且易懂的健康信息的挑战。
• 通过创建一个面向公众的集中式、专家编纂的信息门户，减少错误信息和信息过载。
• 从专家志愿者和信息获取者两个角度，评估专家供稿的AI聊天机器人的有效性。
• 识别可扩展、可信且响应迅速的AI驱动健康信息系统的系统设计启示。
• 探索大型语言模型如何与专家编纂的知识相结合，以增强参与度，同时不损害事实准确性。

提出的方法

• 设计了一种具有层级结构和两阶段验证流程的去中心化专家供稿框架，以确保内容质量和可扩展性。
• 组建了一个由150多位全球范围的卫生与科学专家组成的团队，在疫情期间实时编纂、验证和更新内容。
• 开发了Jennifer，一个AI聊天机器人，通过使用专家验证的、通俗易懂的回答知识库来回答用户查询。
• 对专家志愿者进行了半结构化访谈，以识别其在内容创作与协作中面临的问题与需求。
• 针对信息获取者开展在线实验，评估Jennifer在信息检索、信任度和用户满意度方面的表现。
• 通过每日对话日志建立反馈回路，持续改进聊天机器人的性能与响应能力。

Figure 1 . The figures show the architecture overview of Jennifer . Our expert sourcing framework extends Juji’s base system’s ability to answer people’s questions about COVID-19. Through a hierarchical structure, expert volunteers were divided into four groups, Admins, Curator, Helper, and Tester,

实验结果

研究问题

• RQ1 如何设计专家供稿框架，以在快速演变的危机中支持健康信息的快速、准确和可信编纂？
• RQ2 专家志愿者在参与大规模、实时AI聊天机器人时面临哪些挑战？需要何种技术支持？
• RQ3 专家供稿的AI聊天机器人在多大程度上能够提升非专业用户的信息检索能力、用户满意度和信任度？
• RQ4 如何通过主动设计功能（如通知）支持在长期危机中常见的重复信息获取行为？
• RQ5 大型语言模型在高风险健康信息场景中，如何在保持事实准确性的同时增强用户参与度？

主要发现

• Jennifer成功支持了信息获取者获取准确且相关的信息，用户对其回答表现出高度的满意度和信任度。
• 专家志愿者报告称，缺乏集中化的跟踪系统和实时协作工具是内容更新与协调中的主要挑战。
• 两阶段验证流程与层级团队结构使得多样化专家之间的协作可扩展，同时保持了信息质量。
• 信息获取者频繁返回Jennifer以获取更新信息，表明在长期不确定性期间存在持续且重复的信息需求。
• 该框架在现实世界部署中具有可行性，六个月内实现了数千次用户互动和持续的内容更新。
• 主动通知功能可显著降低用户监控信息演变所需的努力，提示其在未来系统中具有重要价值。

(a) Jennifer opens the conversation with a self-introduction.