[Paper Review] Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

Zihang Dai, Guokun Lai+2 more|arXiv (Cornell University)|Jun 5, 2020

Topic Modeling30 references104 citations

TL;DR

Funnel-Transformer compresses token sequences progressively to reduce computation, while using a decoder to recover token-level representations when needed, improving efficiency with comparable or better performance on sequence-level tasks.

ABSTRACT

With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further improve the model capacity. In addition, to perform token-level predictions as required by common pretraining objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading comprehension. The code and pretrained checkpoints are available at https://github.com/laiguokun/Funnel-Transformer.

Motivation & Objective

• Motivate reducing redundancy from full-length token representations in Transformer-based language models.
• Propose a hierarchical encoder that compresses sequence length across blocks to save FLOPs and memory.
• Demonstrate that saved computation can be reinvested to deepen or widen the model for higher capacity.
• Enable token-level predictions by decoding full-length representations from compressed encodings without altering pretraining objectives.

Proposed method

• Maintain the Transformer backbone but add an encoder that progressively halves sequence length between blocks via pooling.
• Use a pool-query-only attention design where pooling occurs on the query, while keys/values come from the unpooled sequence.
• Apply a simple strided mean pooling (window size 2, stride 2) to halve sequence length at block boundaries.
• Implement a decoder that up-samples the compressed encoder output and fuses it with the first block’s hidden states to recover token-level representations for pretraining objectives.
• Train with MLM or ELECTRA objectives to show generality across pretraining paradigms.
• Discuss complexity benefits: each halving step yields super-linear FLOP reductions, enabling deeper or wider models under similar compute.

Experimental results

Research questions

• RQ1 Can progressively compressing the hidden state sequence achieve similar or better performance with fewer FLOPs than standard Transformers?
• RQ2 How can token-level representations be recovered when the encoder operates on a reduced-length sequence?
• RQ3 Does re-investing the saved FLOPs into depth or width improve model capacity for sequence-level tasks?
• RQ4 How does Funnel-Transformer perform on sequence-level tasks (classification, GLUE, RACE) and on token-level tasks (SQuAD) compared to standard Transformers?

Key findings

• F-TFM often outperforms the standard Transformer under similar or lower FLOPs on GLUE and text classification, especially for smaller models.
• Increasing depth (more blocks) while reducing sequence length yields performance gains due to the extra capacity built with saved computation.
• Partial parameter sharing can hurt performance; in practice, regular layouts with careful design perform best.
• On large-scale pretraining, F-TFM achieves competitive GLUE scores and strong results on RACE, often surpassing baselines under comparable FLOPs.
• When token-level supervision is essential (SQuAD with decoder), very large F-TFM models may still lag behind full-sequence Transformers, highlighting a trade-off for token-level tasks.
• Ablation studies show the pool-query-only design, separate [cls] token handling, and relative positional encoding are important for performance.