[Paper Review] Adversarial Learning for Neural Dialogue Generation

Jiwei Li, Will Monroe+4 more|arXiv (Cornell University)|Jan 23, 2017

Topic ModelingComputer Science45 references221 citations

TL;DR

The paper trains a dialogue generator via adversarial reinforcement learning against a discriminator to produce human-like open-domain responses, and proposes adversarial evaluation as a metric. It shows improvements over standard Seq2Seq baselines on multiple measures.

ABSTRACT

In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator---analagous to the human evaluator in the Turing test--- to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, pushing the system to generate dialogues that mostly resemble human dialogues. In addition to adversarial training we describe a model for adversarial {\em evaluation} that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines.

Motivation & Objective

• Motivate open-domain dialogue generation beyond maximum likelihood training due to dull and repetitive responses.
• Propose an adversarial training framework where a generator learns to produce human-indistinguishable dialogues under discriminator rewards.
• Develop and analyze strategies for providing rewards at each generation step and for reliable evaluation of adversarially trained dialogue systems.
• Investigate whether adversarial training improves interaction quality and how to robustly evaluate such models.

Proposed method

• Formulate dialogue generation as a reinforcement learning problem with a generator G and a discriminator D.
• Use a hierarchical encoder to represent dialogue history and a Seq2Seq-like generator to produce responses.
• Train with policy gradient (REINFORCE) using the discriminator score Q+({x,y}) as a reward for generated utterances.
• Introduce Reward for Every Generation Step (REGS) to assign intermediate rewards via Monte Carlo search or a discriminator designed for partial sequences.
• Incorporate teacher-forcing and alternative reward strategies to stabilize training, including mixing adversarial and MLE updates.
• Pre-train the generator on standard Seq2Seq objectives and the discriminator on real vs. generated data.

Experimental results

Research questions

• RQ1 Does adversarial reinforcement learning produce higher-quality open-domain dialogue responses than standard Seq2Seq training?
• RQ2 How can we reliably evaluate adversarially trained dialogue systems using an automatic evaluator and adversarial metrics?
• RQ3 What reward structures (per-generation-step vs. full-sequence) and training stabilizers best improve conversational quality?
• RQ4 How does adversarial training perform relative to strong baselines (MLE, beam search with MI reranking) on single-turn and multi-turn evaluation?

Key findings

• Adversarially trained models achieve higher quality dialogue responses than standard Seq2Seq baselines according to their evaluations.
• Adversarial Evaluation via Adversary Success (AdverSuc) shows the proposed models outperform baselines in fooling the evaluator, with REGS performing best among the proposed methods.
• Human evaluation indicates significant quality gains for both single-turn and multi-turn dialogues under the adversarial framework.
• Monte Carlo-based intermediate rewards (REGS) yield better performance than vanilla REINFORCE in AdverSup experiments.
• Sampling-based decoding improves AdverSuc but can reduce machine-vs-random discriminability, highlighting evaluation caveats.