[Paper Review] Towards Personalized Dialog Policies for Conversational Skill Discovery

Maryam Fazel-Zarandi, Sampat Biswas+5 more|arXiv (Cornell University)|Nov 15, 2019

Speech and dialogue systems33 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL)-based conversational agent for personalized skill discovery in voice assistants, improving over a rule-based system by adapting to user attributes (e.g., first-time vs. returning users) and conversational styles (e.g., brief vs. verbose). Evaluated in a real production deployment, the RL agent achieves a 76.99% success rate with 4.65 turns on average, significantly outperforming the rule-based agent (73.41% success, 4.97 turns), demonstrating improved personalization and efficiency.

ABSTRACT

Many businesses and consumers are extending the capabilities of voice-based services such as Amazon Alexa, Google Home, Microsoft Cortana, and Apple Siri to create custom voice experiences (also known as skills). As the number of these experiences increases, a key problem is the discovery of skills that can be used to address a user's request. In this paper, we focus on conversational skill discovery and present a conversational agent which engages in a dialog with users to help them find the skills that fulfill their needs. To this end, we start with a rule-based agent and improve it by using reinforcement learning. In this way, we enable the agent to adapt to different user attributes and conversational styles as it interacts with users. We evaluate our approach in a real production setting by deploying the agent to interact with real users, and show the effectiveness of the conversational agent in helping users find the skills that serve their request.

Motivation & Objective

• To address the challenge of skill discovery in large-scale voice assistants where users struggle to find relevant skills among thousands.
• To improve conversational skill discovery by personalizing the interaction based on user attributes and conversational preferences.
• To develop a dialog policy that adapts dynamically to user behavior using reinforcement learning, moving beyond static rule-based systems.
• To evaluate the effectiveness of the RL-based agent in a real production environment with real users.
• To demonstrate that personalized, adaptive dialog policies lead to higher success rates and shorter dialogues.

Proposed method

• Start with a rule-based dialog agent as a baseline for skill discovery in voice assistants.
• Train a reinforcement learning (RL) agent to optimize for both success rate and dialog length, using user feedback as sparse rewards.
• Incorporate user attributes (e.g., first-time vs. returning user) and conversational styles (e.g., concise vs. verbose) as state features in the RL environment.
• Define a state space that includes user context, history, and preferences, and an action space covering recommendations, information requests, and navigation actions.
• Use deep Q-networks (DQN) or similar RL algorithms to learn a policy that maximizes cumulative reward over dialog turns.
• Deploy the trained RL agent in a real production setting to compare performance against the rule-based baseline.

Experimental results

Research questions

• RQ1 Can a reinforcement learning-based dialog policy outperform a rule-based policy in real-world skill discovery for voice assistants?
• RQ2 How does personalization based on user attributes (e.g., first-time vs. returning users) affect dialog success and efficiency?
• RQ3 To what extent does adapting to conversational styles (e.g., brief vs. verbose) improve user satisfaction and task completion?
• RQ4 Does the RL agent learn to reduce dialog length while maintaining or improving success rate?
• RQ5 Can a domain-independent, personalized dialog policy be effectively trained and deployed at scale in production?

Key findings

• The RL-based agent achieved a 76.99% success rate, significantly higher than the rule-based agent’s 73.41% (p < 0.0001).
• The RL agent reduced average dialog length to 4.65 turns, compared to 4.97 for the rule-based agent (p < 0.0001).
• The RL policy outperformed the rule-based policy for both first-time users (77.14% vs. 72.68%) and returning users (76.76% vs. 74.49%), with both differences being statistically significant (p < 0.0001).
• The rule-based policy showed a significant performance gap between first-time and returning users (p = 0.0010), indicating it was biased toward returning users, while the RL policy balanced performance across both groups.
• The RL agent successfully adapted to user attributes and conversational styles, demonstrating the effectiveness of learning personalized policies in real-world settings.
• The study confirms that optimizing for the full dialog interaction—through RL—leads to measurable improvements in both success rate and efficiency.