[论文解读] Trusting the Search: Unraveling Human Trust in Health Information from Google and ChatGPT

Xin Sun, Rongjun Ma+5 more|arXiv (Cornell University)|Mar 15, 2024

Social Media in Health Education被引用 12

一句话总结

本研究比较用户对来自 Google 与 ChatGPT 的健康信息的信任，发现对 ChatGPT 的信任更高，并识别影响信任的因素。

ABSTRACT

People increasingly rely on online sources for health information seeking due to their convenience and timeliness, traditionally using search engines like Google as the primary search agent. Recently, the emergence of generative Artificial Intelligence (AI) has made Large Language Model (LLM) powered conversational agents such as ChatGPT a viable alternative for health information search. However, while trust is crucial for adopting the online health advice, the factors influencing people's trust judgments in health information provided by LLM-powered conversational agents remain unclear. To address this, we conducted a mixed-methods, within-subjects lab study (N=21) to explore how interactions with different agents (ChatGPT vs. Google) across three health search tasks influence participants' trust judgments of the search results as well as the search agents themselves. Our key findings showed that: (a) participants' trust levels in ChatGPT were significantly higher than Google in the context of health information seeking; (b) there is a significant correlation between trust in health-related information and trust in the search agent, however only for Google; (c) the type of search tasks did not affect participants' perceived trust; and (d) participants' prior knowledge, the style of information presentation, and the interactive manner of using search agents were key determinants of trust in the health-related information. Our study taps into differences in trust perceptions when using traditional search engines compared to LLM-powered conversational agents. We highlight the potential role LLMs play in health-related information-seeking contexts, where they excel as stepping stones for further search. We contribute key factors and considerations for ensuring effective and reliable personal health information seeking in the age of generative AI.

研究动机与目标

• 调查在个人健康信息方面的感知信任是否在 Google 与 ChatGPT 之间存在差异。
• 确定在这些搜索代理中影响对健康信息感知信任的因素。
• 检验任务类型、先前知识、信息呈现和互动风格如何影响信任。

提出的方法

• N=21 的同源设计实验室研究，比较 Google 和 ChatGPT 作为健康信息搜索代理。
• 使用自有开放式的健康信息任务，来源于 Yahoo 数据集；参与者可以迭代查询并提出后续提问。
• 任务后调查测量对健康信息以及对搜索代理的信任；半结构化访谈探讨信任因素。
• 对信息的信任度使用经过验证的 11 项量表测量（Cronbach’s alpha .95）。
• 对搜索代理的信任度使用 15 项量表测量（Cronbach’s alpha .62）。
• 数据分析采用双因素重复测量 ANOVA、配对 t 检验、相关分析，以及对访谈的主题分析。

实验结果

研究问题

• RQ1 RQ1: 人们对个人健康相关信息的感知信任是否在 Google 与 ChatGPT 之间存在差异？
• RQ2 RQ2: 在这些不同的搜索代理中，哪些因素促成对个人健康信息的感知信任？

主要发现

• 来自 ChatGPT 的健康信息信任度高于来自 Google。
• 对健康相关信息的信任与对搜索代理的信任之间存在显著相关，但仅对 Google 有效。
• 任务类型对感知信任没有显著影响。
• 先验知识、呈现方式和对搜索代理的互动使用是健康信息信任的关键决定因素。
• 参与者对 ChatGPT 作为搜索代理的总体信任更高（信息的平均信任分数：Google 3.77，ChatGPT 4.05，基于 5 点量表）。
• 对搜索代理的平均信任：Google 3.31，ChatGPT 3.58，基于 5 点量表。