[Paper Review] Adversarial Feature Matching for Text Generation

Yizhe Zhang, Zhe Gan+5 more|arXiv (Cornell University)|Jun 12, 2017

Generative Adversarial Networks and Image SynthesisComputer Science39 references124 citations

TL;DR

TextGAN uses an LSTM generator and CNN discriminator with a kernel-based MMD feature-matching objective to generate realistic sentences and alleviate GAN mode collapse for text data.

ABSTRACT

The Generative Adversarial Network (GAN) has achieved great success in generating realistic (real-valued) synthetic data. However, convergence issues and difficulties dealing with discrete data hinder the applicability of GAN to text. We propose a framework for generating realistic text via adversarial training. We employ a long short-term memory network as generator, and a convolutional network as discriminator. Instead of using the standard objective of GAN, we propose matching the high-dimensional latent feature distributions of real and synthetic sentences, via a kernelized discrepancy metric. This eases adversarial training by alleviating the mode-collapsing problem. Our experiments show superior performance in quantitative evaluation, and demonstrate that our model can generate realistic-looking sentences.

Motivation & Objective

• Motivated by the difficulty of applying GANs to discrete text data and exposure bias in sequence models.
• Aim to generate coherent, diverse sentences by aligning real and synthetic sentence features in a latent space.
• Propose a kernel-based moment-matching objective to stabilize training and encourage variety in outputs.
• Incorporate reconstruction of latent codes to preserve information about the generation process.

Proposed method

• Use an LSTM as the generator to map latent codes z to synthetic sentences.
• Employ a CNN as the discriminator to produce sentence features and a binary real/fake prediction.
• Introduce a feature-matching loss L_MMD^2 to align real and synthetic sentence feature distributions in RKHS with a Gaussian kernel.
• Add a reconstruction loss L_recon to encourage recovery of latent codes from generated sentences.
• Combine GAN loss with L_recon and L_MMD^2 in the discriminator objective to regularize training.
• Use soft-argmax approximation and pre-training strategies to stabilize training and handle discrete text outputs.

Experimental results

Research questions

• RQ1 Can adversarial training with kernel-based feature matching generate realistic sentences more effectively than standard GANs for text?
• RQ2 Does matching the latent feature distributions via MMD alleviate mode-collapse and improve diversity of generated sentences?
• RQ3 How do reconstruction and feature-matching terms affect training stability and generated sentence quality?
• RQ4 What is the impact of embedding and kernel choices on the quality and diversity of generated text?

Key findings

• TextGAN with MMD-based feature matching yields higher BLEU scores and KDE than several baselines, indicating improved sentence quality and diversity.
• MMD and MMD-L variants generally score higher on BLEU and KDE, with MMD-L especially strong on BLEU-2.
• The model demonstrates that synthetic sentence features cover and align with real sentence feature distributions in the 900-dimensional space.
• Pre-training and stabilization techniques (soft-argmax, permutation-based discriminator pre-training) contribute to training stability.
• Compared to AE, VAE, seqGAN, and standard textGAN variants, TextGAN variants achieve better quantitative metrics on the evaluated corpora.