[Paper Review] Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

Natasha Jaques, Asma Ghandeharioun+6 more|arXiv (Cornell University)|Jun 30, 2019

Reinforcement Learning in Robotics64 references131 citations

TL;DR

This paper develops off-policy batch deep RL methods with KL-control and dropout-based uncertainty to learn dialog policies from fixed human-interaction data, achieving improvements over baselines in open-domain dialog without online exploration.

ABSTRACT

Most deep reinforcement learning (RL) systems are not able to learn effectively from off-policy data, especially if they cannot explore online in the environment. These are critical shortcomings for applying RL to real-world problems where collecting data is expensive, and models must be tested offline before being deployed to interact with the environment -- e.g. systems that learn from human interaction. Thus, we develop a novel class of off-policy batch RL algorithms, which are able to effectively learn offline, without exploring, from a fixed batch of human interaction data. We leverage models pre-trained on data as a strong prior, and use KL-control to penalize divergence from this prior during RL training. We also use dropout-based uncertainty estimates to lower bound the target Q-values as a more efficient alternative to Double Q-Learning. The algorithms are tested on the problem of open-domain dialog generation -- a challenging reinforcement learning problem with a 20,000-dimensional action space. Using our Way Off-Policy algorithm, we can extract multiple different reward functions post-hoc from collected human interaction data, and learn effectively from all of these. We test the real-world generalization of these systems by deploying them live to converse with humans in an open-domain setting, and demonstrate that our algorithm achieves significant improvements over prior methods in off-policy batch RL.

Motivation & Objective

• Enable learning from a fixed batch of human interaction data without online exploration in open-domain dialog-based RL.
• Leverage a strong pre-trained prior to regularize policy updates via KL-control to mitigate divergence.
• Use dropout-based uncertainty to lower-bound target Q-values as an alternative to Double Q-Learning.
• Demonstrate learning from implicit human preferences (sentiment, engagement, etc.) through post-hoc reward functions.
• Evaluate robustness and generalization by deploying models live with human users.

Proposed method

• Pre-train a generative model of the state-action space from known trajectories to serve as a prior.
• Train Q-networks with batch data to minimize Bellman-like targets using the fixed batch (Batch Q).
• Introduce dropout-based uncertainty estimates for the target Q-values to obtain a pessimistic lower bound (Batch Q MC).
• Adapt Batch Constrained Q-learning to discrete action spaces using a pre-trained prior (Discrete Batch Constrained Q - DBCQ).
• Incorporate KL-control to penalize divergence from the prior and derive a KL-regularized Q-function that includes prior likelihood and entropy terms (Psi-function formulation).
• Optionally apply model averaging over multiple priors to form a model-averaged prior for more robust learning.

Experimental results

Research questions

• RQ1 Can off-policy batch RL learn effective dialog policies without environment exploration by leveraging a strong pre-trained prior?
• RQ2 Does KL-control from a pre-trained prior stabilize learning and reduce Q-value overestimation in BRL for high-dimensional action spaces like open-domain dialog?
• RQ3 How do dropout-based uncertainty estimates compare to double Q-learning in mitigating overestimation in offline BRL for language generation?
• RQ4 What implicit human reward signals (sentiment, engagement, etc.) can be retrieved post-hoc from interaction data to train dialog policies?
• RQ5 How do KL-control, Psi-learning, and model-averaged priors affect real-world deployment and user-perceived quality of Open-Domain dialog agents?

Key findings

• KL-control models significantly outperform baselines in human evaluation metrics and implicit rewards.
• Dropout-based uncertainty estimates provide a lower-bound Q-value estimate, mitigating overestimation without requiring multiple target networks.
• Model-averaged priors (MA) improve robustness across diverse data sources and architectures.
• Policies that stay close to the pre-trained prior generate more plausible, polite, and engaging dialog than reward-maximizing baselines that exploit rewards.
• Implicit rewards based on sentiment, engagement (words), laughter, semantic similarity, and questioning behavior can guide learning toward higher-quality conversations.
• KL-control with a strong prior is critical for stable learning and improved performance in BRL for open-domain dialog.