[Paper Review] Leveraging Locality in Abstractive Text Summarization

Yixin Liu, Ansong Ni+5 more|arXiv (Cornell University)|May 25, 2022

Topic Modeling4 citations

TL;DR

This paper proposes PageSum, a locality-aware abstractive text summarization model that partitions long documents into non-overlapping pages based on spatial, discourse, or document-level locality, enabling efficient full-attention modeling per page. The model achieves competitive ROUGE scores against strong baselines with efficient attention, demonstrating that locality-based inductive bias can outperform memory-efficient attention approximations in long-text summarization.

ABSTRACT

Neural attention models have achieved significant improvements on many natural language processing tasks. However, the quadratic memory complexity of the self-attention module with respect to the input length hinders their applications in long text summarization. Instead of designing more efficient attention modules, we approach this problem by investigating if models with a restricted context can have competitive performance compared with the memory-efficient attention models that maintain a global context by treating the input as a single sequence. Our model is applied to individual pages which contain parts of inputs grouped by the principle of locality during both encoding and decoding. We empirically investigated three kinds of locality in text summarization at different levels of granularity, ranging from sentences to documents. Our experimental results show that our model has a better performance compared with strong baselines with efficient attention modules, and our analysis provides further insights into our locality-aware modeling strategy.

Motivation & Objective

• To investigate whether models with restricted context can match or exceed the performance of memory-efficient attention models in long-text abstractive summarization.
• To evaluate the effectiveness of different types of locality—spatial, discourse, and document-level—as inductive biases in summarization.
• To reduce the quadratic memory complexity of self-attention without sacrificing performance by leveraging locality in both encoding and decoding.
• To demonstrate that full-attention mechanisms can be preserved within a locality-aware framework, maintaining compatibility with pre-trained models like BART.
• To provide empirical evidence that locality-based modeling can be a viable alternative to efficient attention approximations.

Proposed method

• The input document is divided into non-overlapping pages based on locality principles: sequential proximity (spatial), section boundaries (discourse), or individual documents in a cluster (document-level).
• Each page is encoded independently using a full-attention transformer encoder, preserving the original attention mechanism of models like BART.
• The decoder generates local summaries for each page, producing both a local prediction and a confidence score for that prediction.
• The final summary is generated as a weighted combination of local predictions, using confidence scores to prioritize more reliable outputs.
• The model maintains no cross-page attention during encoding or decoding, enforcing a strict locality inductive bias.
• Three types of locality are evaluated: sentence-level spatial locality, section-level discourse locality, and document-level locality in multi-document settings.

Experimental results

Research questions

• RQ1 Can a locality-aware modeling strategy with restricted context achieve competitive performance compared to efficient attention models in long-text summarization?
• RQ2 Which type of locality—spatial, discourse, or document-level—yields the best performance for different summarization tasks?
• RQ3 Does preserving full-attention mechanisms within local pages outperform efficient attention approximations that maintain global context?
• RQ4 How do long-distance dependencies, where interdependent sentences are far apart, affect the performance of locality-based models?
• RQ5 To what extent does the intrinsic spatial locality in real-world documents support the design of page-based summarization frameworks?

Key findings

• PageSum outperforms strong baselines with efficient attention modules, achieving competitive or better ROUGE scores on the arXiv and GovReport datasets.
• The model shows superior performance on the arXiv dataset, where spatial locality is strong, with a ROUGE-L F1 score of 42.1, outperforming Longformer and BigBird.
• On the GovReport dataset, PageSum achieves a ROUGE-L F1 of 38.5, demonstrating strong performance despite lower spatial locality.
• The model fails to capture long-distance dependencies when two interdependent sentences are far apart, as shown in a case study where only one of two contributing sentences was captured.
• Despite this limitation, long-distance dependencies are relatively rare—only 1.8% of interdependent sentence pairs in the arXiv dataset have a distance ratio above 0.5.
• The results suggest that efficient attention models may not fully realize their design goal of capturing global dependencies, as locality-based models with full attention per page can perform comparably or better.