[Paper Review] Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size

Davis Yoshida, Allyson Ettinger+1 more|arXiv (Cornell University)|Aug 16, 2020

Topic Modeling20 references4 citations

TL;DR

This paper proposes a method to enhance pretrained transformer language models by adding a lightweight recurrence module that enables them to process longer contexts and improve efficiency without increasing memory or computation. By passing a fixed-size representation from one window to the next, the model maintains long-range dependencies and achieves better perplexity than standard fine-tuned GPT-2 on PG-19 and WikiText-103 with the same compute and memory budget.

ABSTRACT

Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years. While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures. We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time. An additional benefit is that our method removes the fixed context size constraint that most transformer models have, allowing for more flexible use. When applied to the GPT-2 language model, we find that our method attains better perplexity than an unmodified GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.

Motivation & Objective

• To reduce the memory and computational cost of fine-tuning pretrained transformers while maintaining or improving performance.
• To overcome the fixed context size limitation of standard transformers, enabling processing of longer sequences.
• To allow existing pretrained models (e.g., GPT-2) to be used more efficiently and flexibly across diverse domains and languages.
• To enable transfer of long-range contextual information across non-overlapping text windows via a learnable recurrence mechanism.
• To provide a plug-and-play method that works with already-trained models without requiring retraining from scratch.

Proposed method

• A recurrence module is added to the fine-tuning process of a pretrained transformer, which computes a fixed-size context representation from the hidden states of each input window.
• This representation is carried forward and used as a memory state during processing of the next window, enabling information flow across windows.
• The recurrence mechanism operates independently of the attention mechanism, preserving the original model’s inductive bias while extending its effective context length.
• The method uses disjoint windows (non-overlapping) during inference, but the recurrence allows information from prior windows to influence current predictions.
• The recurrence module is trained end-to-end with the rest of the model using standard fine-tuning, requiring no architectural changes to the original transformer.
• The approach maintains the same memory footprint as standard fine-tuning and allows a controllable trade-off between computation and perplexity.

Experimental results

Research questions

• RQ1 Can a lightweight recurrence module improve the performance of pretrained transformers without increasing memory or compute usage?
• RQ2 How effectively can a recurrence mechanism propagate long-range dependencies across non-overlapping text windows in a pretrained transformer?
• RQ3 To what extent can this method extend the effective context length of standard transformers beyond their fixed input size?
• RQ4 Does the recurrence mechanism help maintain coherence and reference resolution in long-text generation tasks?
• RQ5 Can this method be applied to existing pretrained models (e.g., GPT-2) to improve performance on long-context tasks without retraining?

Key findings

• The proposed method achieves better perplexity than standard fine-tuned GPT-2 on both PG-19 and WikiText-103 benchmarks, using the same amount of computation and memory.
• The recurrence module successfully propagates high-level topical information and pronoun references across windows, as evidenced by correct pronoun usage in generation samples.
• The model demonstrates improved handling of long-distance dependencies, even when windows are disjoint, due to the learned recurrence mechanism.
• Qualitative analysis shows that the recurrence mechanism acts as a bottleneck, preserving essential information such as character identities and narrative structure.
• The method enables effective use of pretrained models on longer contexts than their original architecture allows, increasing their flexibility and applicability.
• The approach is orthogonal to existing architectural modifications, and could be combined with methods like sparse attention or windowed attention for further gains.