[論文レビュー] BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems

Zachary C. Lipton, Xiujun Li+4 more|arXiv (Cornell University)|Aug 17, 2016

Speech and dialogue systems被引用数 98

ひとこと要約

BBQNはタスク指向対話におけるQ学習に Bayes-by-Backprop-based Thompson sampling を用い、標準的な探索手法を上回り、学習を加速させるためのリプレイバッファスパイクを効果的に実現します。実機およびシミュレーション評価で探索効率の優位性とドメイン拡張適応が示されています。

ABSTRACT

We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as $ε$-greedy, Boltzmann, bootstrapping, and intrinsic-reward-based ones. Additionally, we show that spiking the replay buffer with experiences from just a few successful episodes can make Q-learning feasible when it might otherwise fail.

研究の動機と目的

• Motivate efficient exploration in deep RL for multi-turn task-oriented dialogue systems.
• Propose BBQN, a Bayes-by-Backprop Q-network that uses Thompson sampling for action selection.
• Introduce replay buffer spiking to bootstrap learning from few successful episodes.
• Evaluate BBQN against standard exploration methods in stationary and domain-extension dialogue environments.
• Demonstrate gains via both simulated and real-user evaluations.

提案手法

• Represent Q-function with a Bayesian neural network over weights, enabling a distribution q(w|θ) with Gaussian diagonal posterior.
• Use Thompson sampling to select actions by sampling weights from q and choosing argmax Q(s,a;w).
• Train with a frozen target network and MAP-based targets to improve stability and efficiency.
• Optionally incorporate VIME-style intrinsic rewards (BBQN-VIME) to encourage exploration in uncertain regions.
• Pre-fill the replay buffer with a small set of successful, rule-based experiences to accelerate learning (replay buffer spiking).
• Architectures: MLPs with two 256-node hidden layers, ReLU activations, Adam optimization; 268-dimensional state features; domain-extension handling by adding slots/features progressively.

実験結果

リサーチクエスチョン

• RQ1 Does BBQN improve exploration efficiency over standard DQN exploration strategies in task-oriented dialogue?
• RQ2 How does Bayesian weight uncertainty impact exploration and learning in dialogue policies?
• RQ3 What is the impact of replay buffer spiking on learning speed and final policy performance?
• RQ4 Can BBQN adapt to domain-extension scenarios where new slots become available over time?
• RQ5 How does BBQN compare to intrinsic-reward-based exploration like VIME in both stationary and domain-extension settings?

主な発見

Agent	Full Domain Success Rate	Full Domain Reward	Domain Extension Success Rate	Domain Extension Reward
BBQN-VIME-MAP	0.4856	9.8623	0.6813	15.8223
BBQN-VIME-MC	0.4941	10.4268	0.7120	17.6261
BBQN-MAP	0.5031	10.7093	0.6852	17.3230
BBQN-MC	0.4877	9.9840	0.6722	16.1320
DQN-VIME-MAP	0.3893	5.8616	0.3751	4.9223
DQN-VIME-MC	0.3700	4.9990	0.3675	4.8270
DQN-Bootstrap	0.2516	-0.1300	0.3170	-0.6820
DQN-Boltzmann	0.2658	0.4180	0.2435	-3.4640
DQN	0.2693	0.8660	0.3503	4.7560

• BBQN variants outperform epsilon-greedy, Boltzmann, and bootstrap DQN baselines in both full-domain and domain-extension settings.
• BBQN-MAP achieves the best performance in the full-domain setup, while BBQN-VIME-MC excels in domain-extension scenarios.
• Replay buffer spiking is essential for enabling learning for BBQN and DQN, with benefits saturating beyond a certain number of pre-filled dialogues.
• Real-user evaluations show BBQN significantly outperforms DQN in success rate and user-rated naturalness/coherence after domain extension.
• Across experiments, using MAP targets with Monte Carlo sampling for action selection provides strong performance while maintaining training efficiency.
• BBQN with intrinsic rewards (BBQN-VIME) offers competitive gains, especially in non-stationary environments.