[논문 리뷰] ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

Ruiyun Xu, Yue Feng외 1명|arXiv (Cornell University)|2023. 07. 03.

Misinformation and Its Impacts인용 수 11

한 줄 요약

The study compares ChatGPT-like tools with Google Search-like tools in a randomized online experiment to assess differences in user behavior, perceived information quality, and user experience.

ABSTRACT

The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

연구 동기 및 목표

• Investigate how user behavior differs when using a ChatGPT-like tool versus a Google Search-like tool for information-seeking tasks.
• Assess task performance across education levels for both tools.
• Evaluate perceived information quality, trust, usefulness, enjoyment, and satisfaction across tools.
• Identify potential downsides of ChatGPT, such as overreliance and misinformation.
• Provide insights for search engine design and integration of chatbot technologies.

제안 방법

• Randomized online experiment with two groups: ChatGPT-like tool vs. Google Search-like tool.
• Participants perform information-seeking tasks and may have varying education levels.
• Measure time spent on tasks, overall task performance, and user experience metrics.
• Assess information quality and trust through participant perceptions.
• Analyze potential overreliance and misinformation risks.
• Summarize implications for search engine management and chatbot integration.

실험 결과

연구 질문

• RQ1 Does using a ChatGPT-like tool affect task time compared to a Google Search-like tool for information-seeking tasks?
• RQ2 Is there a difference in overall task performance between the two tools across education levels?
• RQ3 How do users perceive information quality and trust when using ChatGPT-like versus Google Search-like tools?
• RQ4 What are the differences in usefulness, enjoyment, satisfaction, and ease of use between the two tools?
• RQ5 What risks, such as overreliance or misinformation, are associated with ChatGPT-like tools?

주요 결과

• ChatGPT users spend less time on all tasks than Google Search users.
• There is no significant difference in overall task performance between the two groups.
• ChatGPT provides higher perceived information quality but similar trust levels as Google Search.
• ChatGPT users report higher usefulness, enjoyment, and satisfaction, while ease of use is similar across tools.
• ChatGPT may lead to overreliance and generate or replicate misinformation, causing inconsistent results.