[Paper Review] User Intent Recognition and Satisfaction with Large Language Models: A User Study with ChatGPT

Anna Bodonhelyi, Efe Bozkir+3 more|arXiv (Cornell University)|Feb 3, 2024

Topic Modeling4 citations

TL;DR

This user study evaluates intent recognition and satisfaction in GPT-3.5 Turbo and GPT-4 Turbo by applying a fine-grained intent taxonomy and prompt reformulation strategies. Results show GPT-4 outperforms GPT-3.5 on common intents but underperforms on rare ones; users prefer original prompts over reformulated ones despite improved intent recognition, indicating a disconnect between model accuracy and user satisfaction.

ABSTRACT

The rapid evolution of LLMs represents an impactful paradigm shift in digital interaction and content engagement. While they encode vast amounts of human-generated knowledge and excel in processing diverse data types, they often face the challenge of accurately responding to specific user intents, leading to user dissatisfaction. Based on a fine-grained intent taxonomy and intent-based prompt reformulations, we analyze the quality of intent recognition and user satisfaction with answers from intent-based prompt reformulations of GPT-3.5 Turbo and GPT-4 Turbo models. Our study highlights the importance of human-AI interaction and underscores the need for interdisciplinary approaches to improve conversational AI systems. We show that GPT-4 outperforms GPT-3.5 in recognizing common intents but is often outperformed by GPT-3.5 in recognizing less frequent intents. Moreover, whenever the user intent is correctly recognized, while users are more satisfied with the intent-based reformulations of GPT-4 compared to GPT-3.5, they tend to be more satisfied with the models' answers to their original prompts compared to the reformulated ones. The collected data from our study has been made publicly available on GitHub (https://github.com/ConcealedIDentity/UserIntentStudy) for further research.

Motivation & Objective

• To investigate how well GPT-3.5 Turbo and GPT-4 Turbo recognize diverse user intents in real-world prompts.
• To evaluate the impact of intent-based prompt reformulation on user satisfaction with LLM-generated responses.
• To identify performance discrepancies between models across common and rare intent categories.
• To explore user preferences between original prompts and reformulated prompts, especially regarding perceived response quality.
• To inform future LLM development by identifying gaps in intent understanding and user interaction design.

Proposed method

• Developed a fine-grained intent taxonomy based on information retrieval and recent NLP research, categorizing user intents into distinct types such as factual queries, explanatory inquiries, and content creation.
• Collected real user prompts and applied intent-based prompt reformulations using templates derived from the taxonomy to improve clarity and specificity.
• Conducted a controlled user study with participants interacting with both GPT-3.5 and GPT-4 on original and reformulated prompts across various intent categories.
• Measured intent recognition accuracy via model response alignment with predefined intent labels and assessed user satisfaction through post-interaction surveys.
• Incorporated fact-checking references in reformulated prompts for factual queries to enhance credibility and support user verification.
• Analyzed user preferences and satisfaction levels using comparative ratings between original and reformulated prompt responses.

Figure 1: Overview of the proposed prompt reformulation framework, visual summary of Algorithm 1 .

Experimental results

Research questions

• RQ1 How do GPT-3.5 Turbo and GPT-4 Turbo compare in recognizing common and rare user intents?
• RQ2 To what extent does intent-based prompt reformulation improve user satisfaction with LLM responses?
• RQ3 Does improved intent recognition in GPT-4 lead to higher user satisfaction compared to GPT-3.5?
• RQ4 Why do users prefer responses to original prompts even when intent recognition is improved via reformulation?
• RQ5 How does the presence of references in reformulated prompts affect user trust and satisfaction?

Key findings

• GPT-4 Turbo demonstrated significantly higher intent recognition accuracy than GPT-3.5 Turbo across common intent categories such as factual queries, explanatory inquiries, and content creation.
• For rare intent categories like 'curricular planning' and 'learning support', GPT-3.5 Turbo outperformed GPT-4 Turbo, indicating model limitations in handling low-frequency intents.
• Despite higher intent recognition accuracy, users consistently preferred responses to original prompts over those from intent-based reformulations for both models.
• Users showed greater satisfaction with GPT-4’s responses to original prompts than with GPT-3.5’s responses, even when GPT-3.5 received higher satisfaction scores for reformulated prompts.
• The preference for original prompts persisted even when reformulated prompts included references for fact-checking, suggesting users value fluency and naturalness over structured clarity.
• Users demonstrated rapid learning in prompt formulation when provided with reformulation templates, indicating potential for user education in effective prompting.

(a) Confusion matrix for GPT-3.5.