[Paper Review] Extractive Summary as Discrete Latent Variables

Aran Komatsuzaki|arXiv (Cornell University)|Nov 14, 2018

Natural Language Processing Techniques39 references4 citations

TL;DR

This paper proposes using extractive summarization as discrete latent variables for text compression, demonstrating that simple methods like tf-idf and bidirectional language model (bi-LM) loss ranking outperform state-of-the-art VQ-VAE in compressing text. The key finding is that language acts as a strong self-compression code, with a small set of high-informativeness tokens enabling highly effective hierarchical generation.

ABSTRACT

In this paper, we compare various methods to compress a text using a neural model. We find that extracting tokens as latent variables significantly outperforms the state-of-the-art discrete latent variable models such as VQ-VAE. Furthermore, we compare various extractive compression schemes. There are two best-performing methods that perform equally. One method is to simply choose the tokens with the highest tf-idf scores. Another is to train a bidirectional language model similar to ELMo and choose the tokens with the highest loss. If we consider any subsequence of a text to be a text in a broader sense, we conclude that language is a strong compression code of itself. Our finding justifies the high quality of generation achieved with hierarchical method, as their latent variables are nothing but natural language summary. We also conclude that there is a hierarchy in language such that an entire text can be predicted much more easily based on a sequence of a small number of keywords, which can be easily found by classical methods as tf-idf. We speculate that this extraction process may be useful for unsupervised hierarchical text generation.

Motivation & Objective

• To improve unsupervised sequence generation by exploring extractive compression as a method to generate high-quality latent variables.
• To investigate whether extractive methods can outperform learned discrete latent variable models like VQ-VAE in text compression.
• To evaluate the hypothesis that language inherently functions as a compression code, where a few key tokens summarize the rest of the text.
• To explore the potential of extractive methods as a foundation for unsupervised hierarchical text generation.
• To determine whether simple, classical methods like tf-idf can match or exceed complex neural models in identifying informative summary tokens.

Proposed method

• Uses extractive summarization as discrete latent variables by selecting a subset of tokens from the input sequence to represent the full text.
• Compares two extractive methods: selecting tokens with the highest tf-idf scores and selecting tokens with the highest individual loss from a bidirectional language model.
• Employs a hierarchical generation framework where a language model generates the full sequence conditioned on the extracted tokens.
• Evaluates compression performance using conditional perplexity (log-ppl) as the primary metric.
• Trains a bi-LM to compute per-token loss and uses this score to rank and extract the most informative tokens.
• Applies the same evaluation protocol as prior work (e.g., Fan et al., 2018) to ensure fair comparison with state-of-the-art models.

Experimental results

Research questions

• RQ1 Can extractive summarization methods outperform learned discrete latent variable models like VQ-VAE in text compression?
• RQ2 Is there a significant performance gap between simple classical methods (e.g., tf-idf) and complex neural models in identifying informative summary tokens?
• RQ3 To what extent does language act as a self-compression code, where a small number of keywords can predict the rest of the text?
• RQ4 Can extractive methods serve as effective, scalable latent variables for unsupervised hierarchical text generation?
• RQ5 Does the performance of hierarchical generation depend on the quality of the extracted latent representation, and if so, how do different extraction methods compare?

Key findings

• The tf-idf and bi-LM loss-based extraction methods achieved the lowest test log-perplexity (2.32 and 2.36, respectively), significantly outperforming VQ-VAE (3.08).
• The bi-LM method achieved a log-ppl of 2.32, which is 0.52 lower than random selection (2.84), indicating strong sensitivity to informative tokens.
• The performance of tf-idf and bi-LM was nearly identical, suggesting that both methods identify similar, high-informativeness tokens.
• The +tf-idf and +bi-LM models outperformed even the +VQ-VAE and +RL baselines, with the latter performing no better than VQ-VAE.
• The results suggest that language exhibits a hierarchical structure where a small number of keywords contain disproportionately high information, enabling effective compression.
• The study supports the hypothesis that language functions as a strong compression code, where a sequence of keywords can predict the full text with high accuracy, justifying the success of hierarchical generation models.