[Paper Review] Adversarial Text Generation via Feature-Mover's Distance

Li‐Qun Chen, Shuyang Dai+6 more|arXiv (Cornell University)|Sep 17, 2018

Generative Adversarial Networks and Image Synthesis75 citations

TL;DR

Introduces FM-GAN, an adversarial text generation model using feature-m mover’s distance (FMD) based on optimal transport to match real and synthetic sentence features, enabling RL-free training and improved text generation across unconditional, conditional style transfer, and unsupervised decipher tasks.

ABSTRACT

Generative adversarial networks (GANs) have achieved significant success in generating real-valued data. However, the discrete nature of text hinders the application of GAN to text-generation tasks. Instead of using the standard GAN objective, we propose to improve text-generation GAN via a novel approach inspired by optimal transport. Specifically, we consider matching the latent feature distributions of real and synthetic sentences using a novel metric, termed the feature-mover's distance (FMD). This formulation leads to a highly discriminative critic and easy-to-optimize objective, overcoming the mode-collapsing and brittle-training problems in existing methods. Extensive experiments are conducted on a variety of tasks to evaluate the proposed model empirically, including unconditional text generation, style transfer from non-parallel text, and unsupervised cipher cracking. The proposed model yields superior performance, demonstrating wide applicability and effectiveness.

Motivation & Objective

• Motivate and address the challenges of applying GANs to discrete text data.
• Develop a differentiable, OT-based objective (FMD) to match real and generated sentence features.
• Propose an RL-free training framework for text generation with improved stability and diversity.

Proposed method

• Define the Feature-Mover’s Distance (FMD) as a tractable OT-based discrepancy between real and generated sentence features.
• Compute FMD via an IPOT-based optimal transport plan with a cosine-based cost between sentence features.
• Use a CNN-based sentence feature extractor and an LSTM-based sentence generator with a soft-argmax approximation for differentiability.
• Train a generator G and a feature extractor F in a mini-max game using the FMD objective.
• Extend FM-GAN to conditional generation tasks like style transfer and unsupervised decipher via autoencoder-style reconstructions and cycle-consistency concepts.

Experimental results

Research questions

• RQ1 Can an OT-based metric (FMD) provide a stable, discriminative objective for text GANs without reinforcement learning?
• RQ2 Does FM-GAN improve quality and diversity of generated text compared to RL-based and other differentiable GAN approaches?
• RQ3 Can FMD-based adversarial training be effectively extended to conditional generation tasks such as style transfer and deciphering?

Key findings

• FM-GAN yields high-quality text with better diversity in unconditional generation scenarios.
• In non-parallel style transfer, FM-GAN achieves higher sentiment transfer accuracy than several baselines and improves fluency and content preservation.
• The method can be extended to unsupervised deciphering using cycle-consistency and adversarial FMD losses, outperforming some prior variants.
• Compared to Sinkhorn-based OT methods, IPOT-based FMD offers faster convergence and stability without heavy hyperparameter tuning.