[Paper Review] A Hierarchical Neural Autoencoder for Paragraphs and Documents

Jiwei Li, Minh-Thang Luong+1 more|arXiv (Cornell University)|Jun 2, 2015

Topic Modeling29 references207 citations

TL;DR

This paper proposes a hierarchical LSTM autoencoder that encodes paragraphs by modeling word-to-sentence and sentence-to-paragraph relationships, enabling reconstruction of coherent, grammatically correct multi-sentence texts. The model outperforms standard sequence-to-sequence LSTMs, preserving sentence order (L = 1.57 on hotel reviews, L = 2.04 on Wikipedia) and achieving strong ROUGE and Entity Grid scores, demonstrating neural models' capacity to encode discourse coherence.

ABSTRACT

Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models. In this paper, we explore an important step toward this generation task: training an LSTM (Long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs. We introduce an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph. We evaluate the reconstructed paragraph using standard metrics like ROUGE and Entity Grid, showing that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. While only a first step toward generating coherent text units from neural models, our work has the potential to significantly impact natural language generation and summarization\footnote{Code for the three models described in this paper can be found at www.stanford.edu/~jiweil/ .

Motivation & Objective

• To investigate whether neural sequence models can preserve syntactic, semantic, and discourse coherence in long text generation.
• To address the challenge of generating coherent paragraphs and documents using recurrent neural networks.
• To explore if hierarchical compositionality in LSTMs can better encode multi-sentence text structure than flat sequence-to-sequence models.
• To evaluate whether autoencoding long texts can serve as a foundation for more complex generation tasks like summarization or dialogue.

Proposed method

• The model uses a two-level hierarchical LSTM: word-level LSTMs encode individual words into sentence embeddings, and sentence-level LSTMs encode sentences into paragraph-level representations.
• The encoder compresses an input paragraph into a fixed-size vector representation using stacked LSTMs at the word and sentence levels.
• The decoder reconstructs the original paragraph by autoregressively generating words and sentences using the paragraph-level embedding.
• Attention mechanisms are applied at the sentence level to improve alignment between generated and input sentences.
• The model is trained end-to-end using cross-entropy loss to minimize reconstruction error.
• A novel coherence metric, L, evaluates sentence order preservation by measuring the average deviation in relative sentence positions between input and output.

Experimental results

Research questions

• RQ1 Can a hierarchical LSTM autoencoder reconstruct multi-sentence paragraphs with preserved syntactic, semantic, and discourse coherence?
• RQ2 How does modeling hierarchical structure (word → sentence → paragraph) improve text reconstruction compared to flat sequence-to-sequence models?
• RQ3 To what extent can neural models encode discourse-level relationships such as sentence order and coherence?
• RQ4 Does attention at the sentence level improve reconstruction quality and coherence preservation?
• RQ5 Can this autoencoder framework be extended to more complex generation tasks like summarization or question answering?

Key findings

• The hierarchical LSTM model achieved a ROUGE-L F1 score of 0.355 on the hotel-review dataset and 0.220 on Wikipedia, outperforming standard sequence-to-sequence models.
• The model preserved sentence order effectively, with a coherence score L of 1.57 on hotel reviews and 2.04 on Wikipedia, indicating minimal permutation of sentence positions.
• The addition of sentence-level attention improved performance, reducing L to 2.04 on Wikipedia and increasing ROUGE-F1 to 0.291.
• Entity Grid scores showed that the model maintained semantic consistency, with 0.529 F1 on Wikipedia and 0.544 with attention.
• The hierarchical model significantly outperformed standard sequence-to-sequence models in both ROUGE and coherence metrics, demonstrating the benefit of structural inductive bias.
• The results suggest that neural models can encode complex discourse structures, supporting future work in abstractive summarization and dialogue generation.