[Paper Review] SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

Christos Baziotis, Ion Androutsopoulos+2 more|arXiv (Cornell University)|Apr 7, 2019

Topic Modeling38 references22 citations

TL;DR

This paper proposes SEQ^3, a differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression that uses a compressed latent sequence as a bridge between input and reconstructed sentences. By leveraging a pretrained language model prior and a topic-based loss, it achieves state-of-the-art performance without parallel data, outperforming prior unsupervised methods on Gigaword and DUC benchmarks through end-to-end gradient optimization.

ABSTRACT

Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder (SEQ^3), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets.

Motivation & Objective

• To address the lack of parallel training data in abstractive sentence compression by developing a fully differentiable model that does not rely on reinforcement learning.
• To improve unsupervised sentence compression by introducing a sequence-to-sequence-to-sequence architecture with a discrete latent summary as an intermediate representation.
• To enhance summary quality and human readability by incorporating a pretrained language model as a prior over the latent summary sequence.
• To stabilize training and reduce variance in early-stage sampling by introducing a topic-based loss that aligns the compressed summary with input-topic words.
• To demonstrate that end-to-end differentiable training with continuous relaxations can outperform reinforcement learning-based unsupervised models in sentence compression.

Proposed method

• SEQ^3 employs two chained encoder-decoder pairs: a compressor that maps an input sentence to a compressed latent summary, and a reconstructor that reconstructs the original input from the summary.
• The model uses an attentional encoder-decoder architecture with bidirectional LSTMs for encoding the input and generating the summary, and global attention mechanisms to align source and target representations.
• A differentiable relaxation via the Gumbel-Softmax trick enables gradient-based optimization over discrete latent variables, avoiding the instability of reinforcement learning.
• A pretrained language model acts as a prior over the latent summary sequence, encouraging grammatical and fluent outputs through a language modeling loss.
• A topic-based loss is introduced to align the compressed summary with topic-indicating words in the input, reducing training variance and improving semantic consistency.
• The model is trained end-to-end using a composite loss combining reconstruction, language model prior, and topic alignment terms.

Experimental results

Research questions

• RQ1 Can a differentiable sequence-to-sequence-to-sequence autoencoder achieve strong performance in unsupervised abstractive sentence compression without parallel training data?
• RQ2 Does the use of a pretrained language model prior improve the fluency and readability of generated summaries in an unsupervised setting?
• RQ3 Can a topic-based loss stabilize training and improve semantic consistency in abstractive compression without supervision?
• RQ4 How does the proposed model compare to reinforcement learning-based unsupervised models in terms of ROUGE scores and training stability?
• RQ5 To what extent does the model retain content while paraphrasing, and how does it balance extractive and abstractive behaviors?

Key findings

• SEQ^3 outperforms the unsupervised Pretrained Generator of Wang and Lee (2018) across all ROUGE metrics on the Gigaword dataset, achieving state-of-the-art results in unsupervised abstractive sentence compression.
• The removal of the pretrained language model prior significantly degrades performance, especially in ROUGE-2 and ROUGE-L, confirming its role in improving fluency and word order.
• The topic-based loss is critical for training stability; removing it leads to poor convergence and extremely low performance, indicating its role as a bootstrap mechanism.
• On the DUC-2003 and DUC-2004 datasets, SEQ^3 performs competitively, though it does not surpass the prefix baseline on DUC-2003, suggesting limitations in handling certain input types.
• Qualitative analysis shows that SEQ^3 exhibits both extractive and abstractive behaviors: it copies early words when reconstruction is easier, but also paraphrases and compresses multi-word phrases into single content words when appropriate.
• The model demonstrates robustness to distribution shift, generalizing well from Gigaword training data to DUC benchmarks despite no fine-tuning on those datasets.