[Paper Review] Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys.

Ziang Xiao, Michelle X. Zhou+5 more|arXiv (Cornell University)|May 25, 2019

AI in Service Interactions62 references4 citations

TL;DR

This paper proposes an AI-powered chatbot for conducting conversational surveys that uses natural language understanding to interpret free-text responses and dynamically probe for deeper insights. In a field study with 600 participants, the chatbot significantly increased engagement and improved response quality—measured by relevance, depth, and readability—compared to traditional online surveys.

ABSTRACT

The rise of increasingly more powerful chatbots offers a new way to collect information through conversational surveys, where a chatbot asks open-ended questions, interprets a user's free-text responses, and probes answers when needed. To investigate the effectiveness and limitations of such a chatbot in conducting surveys, we conducted a field study involving about 600 participants. In this study, half of the participants took a typical online survey on Qualtrics and the other half interacted with an AI-powered chatbot to complete a conversational survey. Our detailed analysis of over 5200 free-text responses revealed that the chatbot drove a significantly higher level of participant engagement and elicited significantly better quality responses in terms of relevance, depth, and readability. Based on our results, we discuss design implications for creating AI-powered chatbots to conduct effective surveys and beyond.

Motivation & Objective

• To investigate the effectiveness of AI-powered chatbots in conducting conversational surveys compared to traditional online surveys.
• To evaluate participant engagement and response quality when using a chatbot versus a standard survey platform.
• To identify design implications for building chatbots that elicit high-quality, in-depth responses through natural language interaction.
• To understand the limitations and strengths of using NLP-driven chatbots in real-world survey contexts.

Proposed method

• A field study was conducted with approximately 600 participants, half using a standard Qualtrics online survey and the other half interacting with an AI-powered chatbot.
• The chatbot used natural language processing to interpret free-text responses and dynamically generate follow-up questions based on content and context.
• The conversational survey design allowed for open-ended responses with adaptive probing to elicit greater depth and relevance.
• Free-text responses from both conditions were collected and analyzed for engagement, relevance, depth, and readability using qualitative and quantitative metrics.
• The chatbot was designed to maintain conversational flow while ensuring data collection goals were met, mimicking human-like dialogue patterns.

Experimental results

Research questions

• RQ1 How does the engagement level of participants differ between a traditional online survey and a conversational survey conducted via an AI chatbot?
• RQ2 To what extent does the chatbot improve the relevance, depth, and readability of free-text survey responses compared to standard surveys?
• RQ3 What are the key design challenges and opportunities in using AI chatbots for survey data collection?
• RQ4 How do users perceive the conversational flow and interactivity of an AI chatbot in a survey context?

Key findings

• The AI-powered chatbot significantly increased participant engagement compared to the traditional online survey.
• Participants interacting with the chatbot provided responses that were significantly more relevant, deeper in content, and more readable than those in the standard survey condition.
• The chatbot successfully interpreted free-text responses and generated contextually appropriate follow-up questions, enhancing data quality.
• The study revealed that conversational surveys using AI can elicit richer, more nuanced responses than static, linear survey formats.
• Despite its strengths, the chatbot encountered limitations in handling ambiguous or off-topic responses, highlighting the need for improved disambiguation strategies.