[Paper Review] Deep Reinforcement Learning for Dialogue Generation

Jiwei Li, Will Monroe+4 more|arXiv (Cornell University)|Jun 5, 2016

Topic Modeling45 references421 citations

TL;DR

This paper integrates deep reinforcement learning with Seq2Seq dialogue models by simulating two virtual agents and optimizing long-term rewards for informativity, coherence, and ease of answering, leading to more interactive and sustained conversations.

ABSTRACT

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

Motivation & Objective

• Motivate the need to move beyond one-turn MLE-trained Seq2Seq dialogue models toward long-term dialogue success.
• Propose a neural RL generation framework that uses policy gradient to maximize future rewards in simulated dialogues.
• Define reward components capturing forward-looking, informative, and coherent dialogue properties.
• Leverage two-agent dialogue simulations to learn policies that produce more engaging and sustained conversations.

Proposed method

• Represent utterances as actions in an encoder-decoder policy over an infinite action space.
• Simulate dialogues between two virtual agents to explore state-action space and learn a policy p_RL(p_{i+1}|p_i,q_i).
• Define rewards r(a,[p_i,q_i]) combining three terms: ease of answering (r1), information flow (r2), and semantic coherence (r3).
• Train via policy gradient with a curriculum learning strategy that starts with MLE-like tokens and gradually shifts to RL updates.
• Initialize the RL policy with a mutual-information objective and then optimize with policy gradient using a baseline to reduce variance.
• Use AlphaGo-style initialization by pretraining on supervised data and then refining through dialogue simulations.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning with long-term rewards improve open-domain dialogue generation over standard Seq2Seq training?
• RQ2 Do reward components for forward-looking, informative, and coherent dialogue lead to longer, more interactive conversations?
• RQ3 Does a two-agent dialogue simulation framework yield more diverse and sustained responses than traditional approaches?
• RQ4 What is the impact of initializing with mutual information and curriculum learning on RL performance?
• RQ5 How do automatic and human evaluations reflect improvements in long-term dialogue quality?

Key findings

• The RL model yields longer simulated dialogues than Seq2Seq and mutual-information baselines.
• RL-generated responses are more interactive and tend to end with questions, promoting turn-taking.
• RL improves multi-turn quality as judged by humans, and increases ease of answering compared with the baselines.
• Diversity of generated replies is higher under the RL framework than under standard Seq2Seq and mutual information models.
• Mutual information initialization combined with RL leads to the best performance in dialogue sustainment.
• BLEU and perplexity do not correlate with long-term dialogue success; RL shows benefits not captured by these metrics.