[Paper Review] Unsupervised Sentence Compression using Denoising Auto-Encoders

Thibault Févry, Jason Phang|arXiv (Cornell University)|Sep 7, 2018

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a fully unsupervised sentence compression method using a denoising auto-encoder that trains on monolingual text by corrupting sentences with word shuffling and noise, then reconstructing the original. Despite no exposure to compressed sentence pairs, the model generates grammatically correct and meaning-preserving summaries that perform competitively with supervised models in human evaluation, though it underperforms on ROUGE metrics.

ABSTRACT

In sentence compression, the task of shortening sentences while retaining the original meaning, models tend to be trained on large corpora containing pairs of verbose and compressed sentences. To remove the need for paired corpora, we emulate a summarization task and add noise to extend sentences and train a denoising auto-encoder to recover the original, constructing an end-to-end training regime without the need for any examples of compressed sentences. We conduct a human evaluation of our model on a standard text summarization dataset and show that it performs comparably to a supervised baseline based on grammatical correctness and retention of meaning. Despite being exposed to no target data, our unsupervised models learn to generate imperfect but reasonably readable sentence summaries. Although we underperform supervised models based on ROUGE scores, our models are competitive with a supervised baseline based on human evaluation for grammatical correctness and retention of meaning.

Motivation & Objective

• To develop a sentence compression model that does not require parallel pairs of original and compressed sentences.
• To explore whether denoising auto-encoders can learn to compress sentences through unsupervised pretraining on monolingual corpora.
• To evaluate the model’s performance on grammatical correctness and meaning retention without access to target summaries.
• To investigate the impact of sentence embeddings on model output quality and ROUGE scores.
• To assess the limitations of ROUGE as a metric for summarization by comparing it to human evaluation.

Proposed method

• The model uses a denoising auto-encoder architecture trained on corrupted versions of sentences from a single monolingual corpus.
• Input sentences are corrupted via word shuffling and random token noise to simulate a 'noisy' input.
• The model learns to reconstruct the original sentence from the corrupted input, implicitly learning compression through denoising.
• The model is conditioned on a desired output length, allowing generation of summaries of varying compression levels.
• Sentence embeddings (e.g., InferSent) are used to condition the model, though this leads to trade-offs in human evaluation.
• Training is end-to-end and fully unsupervised, relying only on a single monolingual corpus with no parallel data.

Experimental results

Research questions

• RQ1 Can a denoising auto-encoder learn effective sentence compression without any paired training examples?
• RQ2 How does the performance of an unsupervised model compare to a supervised baseline in human evaluation of grammatical correctness and meaning retention?
• RQ3 Does conditioning the model with sentence embeddings improve or degrade the quality of compressed outputs?
• RQ4 To what extent do ROUGE scores correlate with human judgments of summary quality in sentence compression?
• RQ5 Can a model generate meaningful, grammatical summaries by learning to denoise corrupted sentences?

Key findings

• The unsupervised denoising auto-encoder model produces compressed sentences that are grammatically correct and retain the original meaning, performing comparably to a supervised baseline in human evaluation.
• Despite underperforming on ROUGE metrics, especially in F1 scores, the model's outputs are judged as more accurate and natural by human evaluators than those from a supervised model when sentence embeddings are used.
• Incorporating InferSent sentence embeddings improved ROUGE scores but significantly worsened human evaluation scores for both grammar and meaning retention.
• The model generates summaries of varying lengths by conditioning on a desired output length, enabling control over compression level.
• The model learns to compress sentences by recovering structure from shuffled and noisy inputs, suggesting that denoising can implicitly learn compression patterns.
• ROUGE scores were found to be a poor proxy for human evaluation, often failing to correlate with comprehensibility or semantic fidelity.