[Paper Review] Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents

Dookun Park, Yuan Hao+11 more|arXiv (Cornell University)|May 29, 2020

Human Mobility and Location-Based Analysis7 citations

TL;DR

This paper proposes a large-scale hybrid approach that fuses explicit user feedback, machine-learned predictions from user feedback data, and predictions from human-annotated data using a waterfall policy to improve user satisfaction prediction in conversational agents. The method achieves significant performance gains and greater consistency across domains compared to relying solely on human annotations or feedback alone.

ABSTRACT

Measuring user satisfaction level is a challenging task, and a critical component in developing large-scale conversational agent systems serving the needs of real users. An widely used approach to tackle this is to collect human annotation data and use them for evaluation or modeling. Human annotation based approaches are easier to control, but hard to scale. A novel alternative approach is to collect user's direct feedback via a feedback elicitation system embedded to the conversational agent system, and use the collected user feedback to train a machine-learned model for generalization. User feedback is the best proxy for user satisfaction, but is not available for some ineligible intents and certain situations. Thus, these two types of approaches are complementary to each other. In this work, we tackle the user satisfaction assessment problem with a hybrid approach that fuses explicit user feedback, user satisfaction predictions inferred by two machine-learned models, one trained on user feedback data and the other human annotation data. The hybrid approach is based on a waterfall policy, and the experimental results with Amazon Alexa's large-scale datasets show significant improvements in inferring user satisfaction. A detailed hybrid architecture, an in-depth analysis on user feedback data, and an algorithm that generates data sets to properly simulate the live traffic are presented in this paper.

Motivation & Objective

• To address the scalability and representativeness limitations of human-annotated user satisfaction data in large-scale conversational agents.
• To leverage direct user feedback as a more scalable and representative proxy for user satisfaction than manual annotation.
• To overcome incomplete coverage of user feedback in certain intents and domains by combining it with human-annotated predictions.
• To develop a robust, consistent, and scalable user satisfaction prediction system suitable for online experimentation and production deployment.
• To enable self-learning evolution of conversational agents through accurate, real-time satisfaction inference.

Proposed method

• A waterfall policy is used to prioritize prediction sources: first direct user feedback, then feedback-based model predictions (high-confidence), and finally human-annotation-based model predictions.
• Two separate machine-learned models are trained: one on explicit user feedback data and another on human-annotated satisfaction scores.
• The hybrid architecture integrates predictions from both models using a confidence-based routing mechanism to minimize errors in low-coverage scenarios.
• A synthetic data generation algorithm is introduced to simulate live traffic and evaluate model performance under realistic distribution shifts.
• The system is evaluated on a large-scale Alexa dataset, with performance measured using macro-averaged F1, precision, recall, and standard deviation across top 20 domains.
• The approach is designed to be extensible, with plans to enhance the fusion layer using advanced machine learning techniques.

Experimental results

Research questions

• RQ1 Can explicit user feedback significantly improve user satisfaction prediction accuracy compared to human-annotation-only models?
• RQ2 How can a hybrid model that combines user feedback and human annotations achieve better consistency across diverse conversational domains?
• RQ3 What is the impact of varying feedback elicitation rates on model performance and user experience?
• RQ4 How can incomplete feedback coverage in certain intents or domains be mitigated using complementary prediction sources?
• RQ5 Can a hybrid model with a waterfall policy outperform single-source approaches in both accuracy and robustness?

Key findings

• The hybrid approach achieved the highest macro-averaged F1 score, precision, and recall across all evaluated models, significantly outperforming both feedback-only and annotation-only baselines.
• The hybrid model demonstrated the lowest standard deviation in performance across domains, indicating superior consistency in satisfaction prediction.
• Increasing feedback elicitation rates from 1% to 10% led to measurable improvements in prediction accuracy, confirming the scalability and benefit of more feedback data.
• The model maintained high performance even in low-feedback-coverage domains by effectively leveraging the human-annotation-based model as a fallback.
• The waterfall policy effectively prioritized reliable feedback signals, reducing reliance on low-confidence predictions and improving overall robustness.
• The synthetic data generation method successfully simulated real-world traffic patterns, enabling reliable evaluation of model behavior under distribution shifts.