[Paper Review] Generative Cooperative Networks for Natural Language Generation

Sylvain Lamprier, Thomas Scialom+5 more|arXiv (Cornell University)|Jan 28, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces Generative Cooperative Networks (GCN), a novel GAN-based framework for natural language generation that combines a generator and discriminator in a cooperative decoding scheme to stabilize training and ensure theoretical convergence. By using a Boltzmann-like sampling distribution proportional to the product of the previous generator and current discriminator scores, GCN achieves state-of-the-art results in abstractive summarization and question generation, overcoming instability and vanishing gradient issues common in traditional language GANs.

ABSTRACT

Generative Adversarial Networks (GANs) have known a tremendous success for many continuous generation tasks, especially in the field of image generation. However, for discrete outputs such as language, optimizing GANs remains an open problem with many instabilities, as no gradient can be properly back-propagated from the discriminator output to the generator parameters. An alternative is to learn the generator network via reinforcement learning, using the discriminator signal as a reward, but such a technique suffers from moving rewards and vanishing gradient problems. Finally, it often falls short compared to direct maximum-likelihood approaches. In this paper, we introduce Generative Cooperative Networks, in which the discriminator architecture is cooperatively used along with the generation policy to output samples of realistic texts for the task at hand. We give theoretical guarantees of convergence for our approach, and study various efficient decoding schemes to empirically achieve state-of-the-art results in two main NLG tasks.

Motivation & Objective

• To address the instability and lack of convergence in training GANs for discrete sequence generation, particularly in natural language generation (NLG).
• To overcome the limitations of reinforcement learning-based training in language GANs, including high variance and non-stationary rewards.
• To propose a theoretically sound alternative to existing GAN-based NLG methods that avoids catastrophic forgetting and provides convergence guarantees.
• To develop practical, efficient decoding strategies that leverage the discriminator not only for reward shaping but also for cooperative sample generation.
• To achieve state-of-the-art performance in abstractive summarization and question generation using a cooperative, stable, and theoretically grounded training framework.

Proposed method

• Proposes a new training objective where the sampling distribution for imitation is proportional to the product of the previous generator distribution and the current discriminator score: $ q_t \propto p_{t-1} D_t $.
• Introduces a cooperative decoding mechanism that uses the discriminator to guide beam search or Monte Carlo sampling, favoring sequences that are more realistic according to the discriminator.
• Employs importance sampling and reweighting techniques to efficiently optimize the generator using samples from the cooperative distribution.
• Derives theoretical convergence guarantees under assumptions similar to those in continuous GANs (Goodfellow et al., 2014), ensuring the generator distribution converges to the data distribution.
• Adapts the Reward-augmented Maximum Likelihood (RML) framework by Norouzi et al. (2016) to use a learned discriminator as the reward function, improving flexibility over hand-designed metrics.
• Uses a two-stage training process: first train the discriminator on real and generated samples from the current generator, then update the generator using samples from the cooperative distribution $ q_t $.

Experimental results

Research questions

• RQ1 Can a cooperative training scheme between generator and discriminator in discrete GANs ensure theoretical convergence and stability, even with non-stationary rewards?
• RQ2 How can the discriminator be effectively used not only as a reward signal but also as a guide in the decoding process to improve sample quality?
• RQ3 Can a hybrid sampling strategy based on $ p_{t-1} D_t $ outperform standard RL-based or MLE-based training in NLG tasks?
• RQ4 To what extent does the proposed method reduce training variance and avoid catastrophic forgetting compared to prior GAN-based NLG approaches?
• RQ5 Can the theoretical convergence guarantees of continuous GANs be extended to discrete sequence generation via a cooperative framework?

Key findings

• The proposed GCN framework achieves state-of-the-art performance on abstractive summarization and question generation tasks, outperforming both standard MLE and prior GAN-based methods.
• Theoretical analysis shows that under mild assumptions, the generator distribution converges to the true data distribution, extending convergence guarantees from continuous GANs to discrete settings.
• The cooperative decoding strategy, which uses the discriminator to bias sampling, significantly improves sample quality and training stability compared to standard RL fine-tuning.
• The method outperforms MaliGAN (Che et al., 2017) in both stability and performance, even without complex variance reduction techniques.
• Experiments confirm that the sampling distribution $ q_t \propto p_{t-1} D_t $ is more effective than independent negative sampling or pure MLE, especially for long-tailed distributions.
• The approach is robust to model drift and does not suffer from catastrophic forgetting, as the previous generator distribution is preserved in the sampling objective.