[Paper Review] Adversarial Ranking for Language Generation

Kevin Lin, Dianqi Li+3 more|arXiv (Cornell University)|May 31, 2017

Natural Language Processing Techniques24 references158 citations

TL;DR

RankGAN replaces binary discriminators with a ranker in a GAN framework to rank human-written sentences higher than machine-generated ones, training the generator via policy gradient to produce higher-ranked language outputs.

ABSTRACT

Generative adversarial networks (GANs) have great successes on synthesizing data. However, the existing GANs restrict the discriminator to be a binary classifier, and thus limit their learning capacity for tasks that need to synthesize output with rich structures such as natural language descriptions. In this paper, we propose a novel generative adversarial network, RankGAN, for generating high-quality language descriptions. Rather than training the discriminator to learn and assign absolute binary predicate for individual data sample, the proposed RankGAN is able to analyze and rank a collection of human-written and machine-written sentences by giving a reference group. By viewing a set of data samples collectively and evaluating their quality through relative ranking scores, the discriminator is able to make better assessment which in turn helps to learn a better generator. The proposed RankGAN is optimized through the policy gradient technique. Experimental results on multiple public datasets clearly demonstrate the effectiveness of the proposed approach.

Motivation & Objective

• Motivation: improve language generation by moving beyond binary discriminators in GANs.
• Aim: learn from relative ranking information to generate higher-quality natural language.
• Goal: demonstrate RankGAN's effectiveness on multiple public datasets compared to state-of-the-art methods.

Proposed method

• Two-network architecture with a generator G and a ranker R.
• Ranker computes a relative ranking score comparing a candidate sentence to a reference using cosine similarity in embedded space.
• Generator is trained with policy gradient and Monte Carlo rollouts to handle discrete text outputs.
• Ranking score is computed via a softmax-like function over a set of candidates with a reference sentence.
• Training uses a minimax objective that encourages G to produce sentences that rank higher than human-written ones with respect to a reference.
• Ranker training maximizes a ranking objective that contrasts human-written and machine-generated sentences.

Experimental results

Research questions

• RQ1 Can a ranking-based discriminator provide richer feedback than a binary classifier for language generation?
• RQ2 Does RankGAN improve generation quality across diverse language tasks and datasets?
• RQ3 How effective is policy gradient with ranking-based rewards for training text generators?
• RQ4 What impact do reference and comparison set sizes have on RankGAN performance?

Key findings

• RankGAN outperforms SeqGAN and other baselines on synthetic data in terms of negative log-likelihood.
• RankGAN achieves higher BLEU-2/BLEU-3/BLEU-4 scores than baselines on Chinese poems, COCO captions, and Shakespeare data.
• Human evaluation scores favor RankGAN-generated text over SeqGAN on Chinese poems and COCO captions.
• RankGAN demonstrates improved language fluency and diversity inferred from both automatic metrics and human judgments.
• The ranking-based objective is more informative than BLEU-based rewards in guiding generator learning.