[Paper Review] Learning to Encode Text as Human-Readable Summaries using Generative Adversarial Networks

Yau-Shian Wang, Hung-yi Lee|arXiv (Cornell University)|Oct 5, 2018

Topic Modeling22 references4 citations

TL;DR

This paper proposes a GAN-based auto-encoder that learns to compress text into human-readable summaries without requiring document-summary pairs. By training a generator to produce concise, natural-sounding summaries that are both reconstructable by a reconstructor and indistinguishable from real human writing via a discriminator, the model achieves unsupervised abstractive summarization with strong ROUGE scores on both English and Chinese datasets.

ABSTRACT

Auto-encoders compress input data into a latent-space representation and reconstruct the original data from the representation. This latent representation is not easily interpreted by humans. In this paper, we propose training an auto-encoder that encodes input text into human-readable sentences, and unpaired abstractive summarization is thereby achieved. The auto-encoder is composed of a generator and a reconstructor. The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the generator input from the generator output. To make the generator output human-readable, a discriminator restricts the output of the generator to resemble human-written sentences. By taking the generator output as the summary of the input text, abstractive summarization is achieved without document-summary pairs as training data. Promising results are shown on both English and Chinese corpora.

Motivation & Objective

• To develop a text auto-encoder that produces human-readable summaries as latent representations instead of non-interpretable vectors.
• To enable abstractive summarization without relying on document-summary paired training data.
• To improve summary quality by using a discriminator that enforces natural language fluency in generated summaries.
• To explore the feasibility of training sequence-to-sequence models with GANs for discrete text generation, overcoming challenges in training on discrete sequences.
• To evaluate the model’s performance on zero-shot and transfer learning settings across English and Chinese corpora.

Proposed method

• The model uses a generator to encode input text into a short, human-readable summary sequence.
• A reconstructor decodes the generator’s output to reconstruct the original input, minimizing reconstruction loss.
• A discriminator is trained to distinguish between real human-written sentences and the generator’s outputs, encouraging natural language fluency.
• The generator is trained adversarially to fool the discriminator, using a GAN framework to produce more realistic summaries.
• Two GAN training methods—WGAN and adversarial REINFORCE—are evaluated for training stability and performance.
• Transfer learning is explored by pre-training the discriminator on a source domain (e.g., CNN/Daily Mail) and applying it to a target domain (e.g., English Gigaword).

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence auto-encoder generate human-readable summaries without any paired training data?
• RQ2 How effective is adversarial training with a discriminator in improving the fluency and naturalness of generated summaries?
• RQ3 Can the model generalize across domains when using transfer learning with a discriminator pre-trained on a different dataset?
• RQ4 How do different GAN training strategies (e.g., WGAN vs. REINFORCE) affect summary quality and training stability?
• RQ5 To what extent can the generator’s output serve as a meaningful, information-rich summary despite the absence of supervision?

Key findings

• The proposed model achieved competitive ROUGE scores on the Chinese Gigaword dataset, with ROUGE-1 of 41.25, ROUGE-2 of 26.54, and ROUGE-L of 37.76 using adversarial REINFORCE training.
• Even without paired data, the model significantly outperformed the trivial baseline (lead-15) and pre-trained generator, demonstrating the effectiveness of adversarial training.
• The WGAN variant showed faster convergence but slightly lower performance than adversarial REINFORCE on the English Gigaword dataset.
• Transfer learning with a discriminator pre-trained on CNN/Daily Mail improved performance over a pre-trained generator alone, though results were lower than in-domain training due to domain distribution mismatch.
• The generator learned to extract core content and form coherent sentences, though ROUGE-L scores were lower than supervised baselines, likely due to shorter summary length and vocabulary size constraints.
• The model successfully produced summaries that were both reconstructable and human-readable, validating the feasibility of unsupervised abstractive summarization via adversarial training.