[Paper Review] Cross-Document Language Modeling.

Avi Caciularu, Arman Cohan+4 more|arXiv (Cornell University)|Jan 2, 2021

Topic Modeling14 citations

TL;DR

This paper proposes Cross-Document Language Modeling (CD-LM), a pretraining approach that enhances masked language modeling for multi-document NLP tasks by using cross-document masking and a novel long-context attention pattern with sequence-level global attention. CD-LM achieves state-of-the-art results on tasks like coreference resolution and plagiarism detection with significantly fewer parameters than prior methods.

ABSTRACT

We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our cross-document language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related documents in a single input, via cross-document masking, which encourages the model to learn cross-document and long-range relationships. Second, extending the recent Longformer model, we pretrain with long contexts of several thousand tokens and introduce a new attention pattern that uses sequence-level global attention to predict masked tokens, while retaining the familiar local attention elsewhere. We show that our CD-LM sets new state-of-the-art results for several multi-text tasks, including cross-document event and entity coreference resolution, paper citation recommendation, and documents plagiarism detection, while using a significantly reduced number of training parameters relative to prior works.

Motivation & Objective

• To improve masked language modeling for multi-document NLP tasks by capturing long-range and cross-document dependencies.
• To reduce the number of training parameters required for effective multi-document modeling compared to prior approaches.
• To extend the Longformer architecture with a new attention pattern that supports sequence-level global attention for better context modeling.
• To enhance performance on downstream multi-document tasks such as coreference resolution and citation recommendation.

Proposed method

• Pretrain the model on multiple related documents in a single input sequence, enabling cross-document masking to learn inter-document relationships.
• Use a modified Longformer attention mechanism that incorporates sequence-level global attention to predict masked tokens across long contexts.
• Apply local attention within local windows while maintaining global attention over the entire sequence to preserve long-range dependencies.
• Train the model end-to-end on large-scale multi-document corpora using masked language modeling objective.
• Optimize the model with a new attention pattern that balances computational efficiency and long-context modeling.
• Fine-tune the pre-trained CD-LM on specific multi-document downstream tasks to evaluate performance.

Experimental results

Research questions

• RQ1 Can cross-document masking improve masked language modeling for multi-document NLP tasks?
• RQ2 How does incorporating sequence-level global attention in long-context modeling enhance performance on multi-document tasks?
• RQ3 To what extent can CD-LM reduce parameter count while maintaining or improving performance compared to prior multi-document models?
• RQ4 Does the combination of multi-document pretraining and long-context attention lead to better generalization on coreference and plagiarism detection tasks?

Key findings

• CD-LM achieves state-of-the-art results on cross-document event and entity coreference resolution tasks.
• The model outperforms prior methods on paper citation recommendation, demonstrating improved relevance prediction.
• CD-LM achieves strong performance in document plagiarism detection, indicating effective modeling of cross-document semantic similarity.
• The model uses significantly fewer training parameters than previous approaches while maintaining or exceeding performance.
• The integration of cross-document masking and sequence-level global attention leads to measurable improvements in long-range dependency learning.
• The ablation study confirms that both cross-document masking and the new attention pattern contribute independently to performance gains.