[Paper Review] Efficient Long-Text Understanding with Short-Text Models

Maor Ivgi, Uri Shaham+1 more|arXiv (Cornell University)|Aug 1, 2022

Topic Modeling4 citations

TL;DR

This paper proposes SLED, a method that enables off-the-shelf short-text pretrained encoder-decoder models (e.g., BART, T5) to efficiently understand long texts by partitioning input into overlapping chunks, encoding each independently, and fusing representations via the pretrained decoder. SLED achieves competitive performance—matching or exceeding specialized long-context models like LongT5 base and UL2, despite using models up to 50x smaller and requiring no additional pretraining.

ABSTRACT

Transformer-based pretrained language models (LMs) are ubiquitous across natural language understanding, but cannot be applied to long sequences such as stories, scientific articles and long documents, due to their quadratic complexity. While a myriad of efficient transformer variants have been proposed, they are typically based on custom implementations that require expensive pretraining from scratch. In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). We illustrate through controlled experiments that SLED offers a viable strategy for long text understanding and evaluate our approach on SCROLLS, a benchmark with seven datasets across a wide range of language understanding tasks. We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.

Motivation & Objective

• Address the challenge of applying standard pretrained language models to long sequences, which are limited by quadratic computational complexity.
• Overcome the need for expensive, custom pretraining of specialized long-context models by reusing existing, off-the-shelf short-text models.
• Develop a simple, generalizable framework that enables strong long-text performance without architectural modifications or additional pretraining.
• Provide a diagnostic tool to analyze the long-range dependency requirements of different NLP benchmarks.
• Demonstrate that fusion-in-decoder with chunked encoding can effectively handle long-range reasoning in real-world tasks.

Proposed method

• Partition long input sequences into overlapping chunks of fixed length to maintain local context continuity.
• Encode each chunk independently using a pretrained encoder from an off-the-shelf encoder-decoder LM (e.g., BART, T5).
• Feed all encoded chunk representations into a pretrained decoder that attends to all chunks to perform cross-chunk fusion.
• Use the decoder’s autoregressive generation capability to produce final outputs based on fused representations, mimicking fusion-in-decoder (FiD) but adapted for long documents.
• Leverage the positional encoding of the backbone model independently per chunk, enabling generalization to arbitrary input lengths.
• Ensure robustness for edge tokens by using overlapping chunks and defining effective chunk tokens to preserve context across boundaries.

Experimental results

Research questions

• RQ1 Can a simple, modular framework reusing existing short-text models achieve competitive long-text understanding without specialized pretraining?
• RQ2 How effective is fusion-in-decoder with chunked encoding for long-range reasoning compared to dedicated long-context architectures?
• RQ3 To what extent do long-text benchmarks like SCROLLS rely on remote, non-local context, and which tasks require deeper cross-chunk reasoning?
• RQ4 How does SLED’s performance scale with model size and input length, and how does it compare to state-of-the-art specialized models?
• RQ5 Can SLED serve as a diagnostic tool to analyze the long-range dependency requirements of different NLP tasks?

Key findings

• SLED improves the average SCROLLS score of pretrained models by 6 points across multiple tasks, demonstrating strong generalization on long-context benchmarks.
• BART large - SLED achieves performance comparable to LongT5 base, a model specifically pretrained for long-range dependencies, despite having only 1/50th the parameters.
• SLED outperforms UL2, a 50x larger model, on the SCROLLS benchmark, showing that parameter efficiency does not compromise performance when leveraging effective architectural reuse.
• SLED’s performance is especially strong on datasets with limited training data, where it significantly outperforms LED, which underperforms due to poor generalization on small long-context datasets.
• Controlled experiments confirm that SLED can locate relevant information within long sequences and effectively fuse information from independently encoded chunks.
• Analysis using SLED reveals that QA and NLI tasks in SCROLLS require relatively local contextualization, while summarization and certain NLI tasks benefit more from long-range fusion.