[Paper Review] Language Model Pre-training for Hierarchical Document Representations

Ming‐Wei Chang, Kristina Toutanova+2 more|arXiv (Cornell University)|Jan 26, 2019

Topic Modeling8 references18 citations

TL;DR

This paper proposes novel pre-training methods for hierarchical document representations that integrate contextual information from entire documents, using masked language modeling and bidirectional sequence modeling to improve performance on document-level NLP tasks. Experiments show significant gains on document segmentation, question answering, and extractive summarization, with improvements of up to 6% absolute on TriviaQA and strong results without reinforcement learning.

ABSTRACT

Hierarchical neural architectures are often used to capture long-distance dependencies and have been applied to many document-level tasks such as summarization, document segmentation, and sentiment analysis. However, effective usage of such a large context can be difficult to learn, especially in the case where there is limited labeled data available. Building on the recent success of language model pretraining methods for learning flat representations of text, we propose algorithms for pre-training hierarchical document representations from unlabeled data. Unlike prior work, which has focused on pre-training contextual token representations or context-independent {sentence/paragraph} representations, our hierarchical document representations include fixed-length sentence/paragraph representations which integrate contextual information from the entire documents. Experiments on document segmentation, document-level question answering, and extractive document summarization demonstrate the effectiveness of the proposed pre-training algorithms.

Motivation & Objective

• To address the challenge of learning effective hierarchical document representations with limited labeled data by leveraging large-scale unlabeled text.
• To overcome the limitations of existing pre-training methods that focus only on local context or unidirectional representations.
• To enable bidirectional fusion of long-range contextual information in hierarchical document models through novel pre-training objectives.
• To evaluate the impact of pre-training both local (sentence) and global (document) levels of hierarchical representations on downstream tasks.
• To demonstrate that pre-training the higher-level document representations yields larger performance gains than pre-training only lower-level representations.

Proposed method

• Proposes a hierarchical neural architecture with two levels: word-level representations contextualized within sentences, and sentence-level representations contextualized within the full document.
• Introduces a left-to-right and right-to-left language modeling approach to pre-train unidirectional hierarchical representations, extending the ELMo-style method to document-level context.
• Develops a masked language modeling objective specifically adapted for hierarchical representations, enabling efficient pre-training of bidirectional document-level context.
• Uses a two-stage training process: first pre-train the hierarchical model on large-scale unlabeled documents using the proposed objectives, then fine-tune on task-specific labeled data with a lightweight task network.
• Applies the pre-trained representations as fixed features or fine-tunes the full model on downstream tasks such as document segmentation, passage retrieval, and extractive summarization.
• Employs ROUGE scores and passage retrieval accuracy to evaluate performance, comparing against strong baselines including ELMo and skip-thought vectors.

Experimental results

Research questions

• RQ1 Can pre-training hierarchical document representations from large-scale unlabeled text improve performance on document-level NLP tasks?
• RQ2 Does pre-training the higher-level (document-level) representations yield greater gains than pre-training only the lower-level (sentence-level) representations?
• RQ3 How does bidirectional context integration in hierarchical representations affect performance compared to unidirectional or isolated sentence representations?
• RQ4 Can pre-trained hierarchical representations outperform strong baselines like ELMo and skip-thought vectors without task-specific fine-tuning or reinforcement learning?
• RQ5 What is the relative contribution of local (sentence) and global (document) contextualization in hierarchical pre-training for extractive summarization and question answering?

Key findings

• The proposed masked language modeling approach for hierarchical representations, { extsc{mask-LM}}$^{\text{global}}$, achieves the best performance on the TriviaQA passage retrieval task, improving over prior work by 6% absolute.
• On the CNN/Daily Mail summarization task, the model achieves strong results without using reinforcement learning, outperforming prior extractive summarization methods by 0.9 ROUGE-1 and 0.7 ROUGE-L points.
• Pre-training the higher-level document representations leads to larger performance gains than pre-training only the lower-level sentence representations in most tasks.
• The model outperforms strong baselines such as ELMo and skip-thought vectors when used as fixed features in document-level Bi-LSTM models, demonstrating the superiority of global hierarchical pre-training.
• The impact of pre-training is most pronounced when both local and global representations are pre-trained, confirming the value of multi-level contextualization.
• The results suggest that bidirectional hierarchical representations pre-trained with masked language modeling are highly effective and complementary to auto-regressive generation models in abstractive summarization.