[Paper Review] Budgeted Policy Learning for Task-Oriented Dialogue Systems

Zhirui Zhang, Xiujun Li+2 more|arXiv (Cornell University)|Jun 2, 2019

Speech and dialogue systems30 references19 citations

TL;DR

This paper proposes BCS-DDQ, a budget-conscious scheduling framework that enhances sample efficiency in task-oriented dialogue systems by intelligently allocating a fixed number of real user interactions. By integrating a Poisson-based scheduler, active user goal sampling, and a dynamic controller, BCS-DDQ outperforms SOTA baselines in success rate on a movie-ticket booking task with limited human interaction budgets.

ABSTRACT

This paper presents a new approach that extends Deep Dyna-Q (DDQ) by incorporating a Budget-Conscious Scheduling (BCS) to best utilize a fixed, small amount of user interactions (budget) for learning task-oriented dialogue agents. BCS consists of (1) a Poisson-based global scheduler to allocate budget over different stages of training; (2) a controller to decide at each training step whether the agent is trained using real or simulated experiences; (3) a user goal sampling module to generate the experiences that are most effective for policy learning. Experiments on a movie-ticket booking task with simulated and real users show that our approach leads to significant improvements in success rate over the state-of-the-art baselines given the fixed budget.

Motivation & Objective

• To address the challenge of limited real user interactions in training effective task-oriented dialogue agents.
• To reduce the discrepancy between simulated and real user behaviors in policy learning.
• To improve sample efficiency in reinforcement learning by strategically selecting high-impact training experiences.
• To integrate human demonstrations and active learning into the Deep Dyna-Q framework for better policy optimization under budget constraints.
• To empirically validate the effectiveness of the proposed framework in both simulated and real-user settings.

Proposed method

• A Poisson-based global scheduler dynamically allocates the fixed budget of real user interactions across training stages.
• An active user goal sampling module selects unexplored or previously failed user goals to generate high-impact training experiences.
• A controller decides at each step whether to collect real human-human conversations, conduct human-agent interactions, or generate simulated experiences via the world model.
• The world model is trained on real experiences and used to generate high-quality simulated interactions for indirect reinforcement learning.
• The dialogue policy is updated via both direct reinforcement learning (on real data) and indirect planning (on simulated data).
• The framework extends Deep Dyna-Q with budget-conscious scheduling to improve sample efficiency and policy performance under fixed interaction budgets.

Experimental results

Research questions

• RQ1 How can a fixed, small number of real user interactions be optimally allocated across different stages of dialogue policy training?
• RQ2 What user goals are most effective for improving policy performance, and how can they be actively selected?
• RQ3 How does integrating human demonstrations into the training process improve sample efficiency in low-budget settings?
• RQ4 To what extent does the proposed budget-conscious scheduling mechanism outperform standard RL and DDQ baselines under fixed interaction budgets?
• RQ5 What is the relative contribution of active sampling versus budget allocation in enhancing policy learning efficiency?

Key findings

• BCS-DDQ significantly improves success rate over SOTA baselines, including DQN and DDQ, under a fixed budget of 200 real user interactions.
• The BCS-DDQ agent successfully books tickets in 100% of test cases, while the DDQ agent fails due to repeated, unproductive inquiries about theater availability.
• The ablation study shows that active sampling contributes more to performance than budget allocation, with performance dropping significantly when random sampling replaces active sampling.
• The learning curve of BCS-DDQ consistently outperforms both DQN and DDQ throughout training, confirming sustained improvement.
• The controller in BCS-DDQ effectively balances real interaction, human demonstration, and simulated experience use, leading to faster convergence and higher success rates.
• Human-in-the-loop experiments confirm that BCS-DDQ produces more natural, correct, and task-completing dialogues than DDQ.