[Paper Review] Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

Yi Tay, Mostafa Dehghani+8 more|arXiv (Cornell University)|Sep 22, 2021

Topic ModelingComputer Science50 references58 citations

TL;DR

The paper empirically studies Transformer scaling for pretraining and fine-tuning, showing that model shape matters for downstream transfer, scaling effects vary by compute region, and proposing DeepNarrow scaling for Pareto-efficient models with fewer params and faster training.

ABSTRACT

There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.

Motivation & Objective

• Assess how upstream pretraining scaling relates to downstream transfer performance.
• Investigate how model shape (depth vs width) affects fine-tuning outcomes across tasks.
• Characterize scaling behavior across different compute regimes and model sizes.
• Identify practical, Pareto-efficient scaling strategies for transfer learning with Transformers.
• Provide pretrained checkpoints and tooling to facilitate future scaling research.

Proposed method

• Use encoder-decoder Transformer with relative attention based on T5 architecture for a wide range of sizes (tiny to XXXL).
• Pretrain models on Colossal Cleaned Common Crawl (C4) using span-based MLM with 2^19 steps on TPU-v3 hardware.
• Fine-tune on 17 downstream tasks including GLUE, SuperGLUE, and SQuAD, reporting aggregate SuperGLUE accuracy.
• Systematically vary scaling operators (depth, width, hidden size, KV, heads, etc.) and measure upstream perplexity and downstream transfer.
• Analyze Pareto-frontier of configurations to assess efficiency in terms of parameters, FLOPs, and throughput.
• Publicly release over 100 pretrained checkpoints and conduct cross-domain checks on Vision Transformers (ViT).

Experimental results

Research questions

• RQ1 Does scaling behavior observed in upstream pretraining generalize to downstream transfer in a pretrain-finetune setting?
• RQ2 How does model shape (depth vs width) influence downstream transfer performance across tasks?
• RQ3 Do scaling strategies yield the same efficiency across different compute regions (small vs large) and modalities?
• RQ4 Can we derive practical scaling protocols that improve Pareto efficiency without sacrificing downstream quality?
• RQ5 Are findings consistent across NLP tasks and transferable to vision models like ViT?

Key findings

• Downstream transfer performance depends strongly on model shape, not just parameter count, contrasting upstream trends.
• Pretraining perplexity often misleadingly predicts downstream quality; upstream gains do not always transfer to downstream tasks.
• Canonical sizes like T5-base/Large are Pareto-inefficient compared to well-chosen alternative configurations.
• Scaling effects differ across compute regimes; strategies effective at small scale may not generalize to larger compute regions.
• DeepNarrow scaling (favoring depth before width) yields Pareto-efficient models with similar or better downstream performance while using fewer parameters and training faster; this approach also transfers to ViT and to additional NLP tasks beyond GLUE/SuperGLUE/SQuAD.