[Paper Review] Sample-efficient Deep Reinforcement Learning for Dialog Control

Kavosh Asadi, J. D. Williams|arXiv (Cornell University)|Dec 18, 2016

Reinforcement Learning in Robotics12 references10 citations

TL;DR

This paper proposes three sample-efficient deep reinforcement learning methods for dialog control using recurrent neural networks (RNNs). By introducing a value network to reduce policy gradient variance and applying experience replay to both policy and value networks, the approach reduces the number of required dialogs by approximately one-third compared to standard policy gradient methods, with consistent gains across both partially observable dialog and fully observable Lunar Lander tasks.

ABSTRACT

Representing a dialog policy as a recurrent neural network (RNN) is attractive because it handles partial observability, infers a latent representation of state, and can be optimized with supervised learning (SL) or reinforcement learning (RL). For RL, a policy gradient approach is natural, but is sample inefficient. In this paper, we present 3 methods for reducing the number of dialogs required to optimize an RNN-based dialog policy with RL. The key idea is to maintain a second RNN which predicts the value of the current policy, and to apply experience replay to both networks. On two tasks, these methods reduce the number of dialogs/episodes required by about a third, vs. standard policy gradient methods.

Motivation & Objective

• To improve sample efficiency in deep reinforcement learning for dialog policy optimization, where each dialog interaction is costly.
• To address the high variance and sample inefficiency of standard policy gradient methods when training RNN-based dialog policies.
• To enable effective training of RNN policies using fewer episodes by leveraging value function estimation and experience replay.
• To evaluate the method on both partially observable dialog tasks and a fully observable control task (Lunar Lander) for generalization.
• To demonstrate that the method remains effective across different network architectures, optimizers, and task types.

Proposed method

• Introduces a value network $\hat{V}(\mathbf{h}_t, \mathbf{w})$ to estimate the expected return from each hidden state $\mathbf{h}_t$, used as a state-dependent baseline to reduce gradient variance in policy updates.
• Applies experience replay to both the policy network and value network, allowing multiple gradient updates per episode and improving data efficiency.
• Uses off-policy updates via importance sampling ratios $\rho_t = \pi(a_t|\mathbf{h}_t)/\mu(a_t|\mathbf{h}_t)$ to enable experience replay with non-i.i.d. data from behavior policy $\mu$.
• Employs a two-stream training process: on-policy updates for the policy network and off-policy updates for the value network, with multiple off-policy updates per on-policy update.
• Uses stochastic gradient descent with adaptive optimizers (Adadelta for dialog, Adam for Lunar Lander) and batch sizes tuned per task.
• Applies a recurrent architecture (LSTM or GRU) to model the policy and value functions, enabling handling of partial observability in dialog settings.

Experimental results

Research questions

• RQ1 Can a value network reduce the sample complexity of policy gradient training in RNN-based dialog policies?
• RQ2 To what extent does experience replay improve sample efficiency when applied to both policy and value networks in dialog control?
• RQ3 How does the proposed method perform on a non-dialog, fully observable RL task like Lunar Lander, indicating generalization potential?
• RQ4 Does the method maintain performance gains when changing network architecture, activation functions, and optimizers?
• RQ5 Can the method reduce training variance and accelerate convergence compared to standard on-policy policy gradient methods?

Key findings

• The proposed method reduced the number of dialogs required to reach asymptotic performance by approximately one-third compared to standard on-policy policy gradient methods in the dialog control task.
• The method achieved lower variance in task success across 200 independent runs, indicating more stable learning.
• On the Lunar Lander task, the method showed a similar improvement in sample efficiency, confirming its robustness across different environments and network architectures.
• The use of a value network as a baseline significantly reduced gradient variance, leading to faster and more stable convergence.
• Experience replay enabled multiple gradient updates per episode, increasing data efficiency and reducing the need for new interactions.
• The method generalized well across different optimizers (Adadelta vs. Adam), activation functions (ReLU), and network designs, suggesting broad applicability.