[Paper Review] Are Pre-trained Convolutions Better than Pre-trained Transformers?

Yi Tay, Mostafa Dehghani+5 more|arXiv (Cornell University)|May 7, 2021

Topic Modeling42 references33 citations

TL;DR

The paper empirically evaluates pre-trained convolutional Seq2Seq models against pre-trained Transformers across 8 NLP tasks, finding that CNN-based models are competitive and often outperform Transformers in several settings, and that pre-training benefits CNNs similarly to Transformers.

ABSTRACT

In the era of pre-trained language models, Transformers are the de facto choice of model architectures. While recent research has shown promise in entirely convolutional, or CNN, architectures, they have not been explored using the pre-train-fine-tune paradigm. In the context of language models, are convolutional models competitive to Transformers when pre-trained? This paper investigates this research question and presents several interesting findings. Across an extensive set of experiments on 8 datasets/tasks, we find that CNN-based pre-trained models are competitive and outperform their Transformer counterpart in certain scenarios, albeit with caveats. Overall, the findings outlined in this paper suggest that conflating pre-training and architectural advances is misguided and that both advances should be considered independently. We believe our research paves the way for a healthy amount of optimism in alternative architectures.

Motivation & Objective

• Assess whether convolutional architectures can match or beat Transformer-based models under the pre-train–fine-tune paradigm.
• Evaluate multiple convolution variants (lightweight, dilated, dynamic) across diverse NLP tasks.
• Measure the impact of pre-training on CNNs versus Transformers in terms of accuracy, speed, and scalability.

Proposed method

• Develop a pre-trained convolutional Seq2Seq model using span-based sequence-to-sequence denoising objectives inspired by T5.
• Investigate several convolution variants: lightweight depthwise convolutions, dynamic convolutions, and dilated convolutions.
• Pre-train both CNNs and Transformers on a large corpus (C4) for 524K steps with a 3-token span corruption, then fine-tune on downstream tasks.
• Evaluate on 8 datasets covering toxicity detection, sentiment, news classification, question classification, and compositional generalization.
• Use 12-layer encoder–decoder architectures with 768 model dimension and 12 heads; compare against a Transformer baseline (T5) of similar scale.

Experimental results

Research questions

• RQ1 Do convolutions benefit from pre-training as much as Transformers?
• RQ2 Are pre-trained convolutional models competitive with pre-trained Transformers, and in which tasks and settings do they excel or fail?
• RQ3 What practical benefits do pre-trained convolutions offer over Transformers in terms of speed, FLOPs, and scalability?
• RQ4 What are the caveats and failure modes of pre-trained convolutions, and when should one avoid them?
• RQ5 Are certain convolution variants (lightweight, dynamic, dilated) generally better than others under pre-training?

Key findings

• Pre-trained convolutions are competitive with Transformers and can outperform them on several tasks.
• Across 8 datasets, pre-trained convolutions outperform a state-of-the-art Transformer (T5) on 6 of 7 tasks, with notable gains in some settings.
• Convolutions are faster and more FLOP-efficient than Transformers and scale better to longer sequences.
• Pre-training benefits CNNs similarly to Transformers, indicating that pre-training advantages are not exclusive to Transformer architectures.
• Among convolution variants, dilated and dynamic convolutions generally perform better than lightweight convolutions in many settings.
• Cross-attention limitations in CNNs can hinder tasks requiring inter-sequence interactions, but a single cross-attention layer can restore much of this gap.