[Paper Review] Pretrained Transformers as Universal Computation Engines

Kevin Lü, Aditya Grover+2 more|arXiv (Cornell University)|Mar 9, 2021

Ferroelectric and Negative Capacitance DevicesEngineering60 references99 citations

TL;DR

A GPT-2 style language transformer can be frozen (self-attention and feedforward layers) and fine-tuned only input/output layers plus layer norm to perform across modalities (numerical, vision, protein) with competitive accuracy and faster convergence, suggesting language pretraining endows universal computation capabilities.

ABSTRACT

We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning -- in particular, without finetuning of the self-attention and feedforward layers of the residual blocks. We consider such a model, which we call a Frozen Pretrained Transformer (FPT), and study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. In contrast to prior works which investigate finetuning on the same modality as the pretraining dataset, we show that pretraining on natural language can improve performance and compute efficiency on non-language downstream tasks. Additionally, we perform an analysis of the architecture, comparing the performance of a random initialized transformer to a random LSTM. Combining the two insights, we find language-pretrained transformers can obtain strong performance on a variety of non-language tasks.

Motivation & Objective

• Investigate whether a transformer pretrained on natural language can generalize to other modalities with minimal finetuning.
• Assess the role of the pretraining modality versus architecture in cross-domain transfer.
• Evaluate the importance of freezing self-attention and feedforward layers and finetuning only peripheral components.
• Compare Transformer versus LSTM baselines in cross-modal tasks.
• Analyze compute efficiency gains from language pretraining for downstream tasks.

Proposed method

• Use a frozen pretrained GPT-2 transformer as a universal computation engine (FPT) by freezing self-attention and feedforward layers.
• Fine-tune only the input embedding layer, the output layer, and layer norm (and optionally positional embeddings) for diverse downstream tasks.
• Evaluate on seven classification tasks spanning numerical computation, image classification, and protein fold prediction.
• Compare against fully trained transformers and LSTMs, and against other pretraining modalities (Bit memory, ViT).
• Analyze attention patterns, convergence speed, and ablations to identify what is driving transfer.

Experimental results

Research questions

• RQ1 Can a language-pretrained transformer transfer to different modalities without updating its core attention/FFN parameters?
• RQ2 How important is the pretraining modality (language vs random vs image) for cross-modal transfer?
• RQ3 Is the transformer architecture essential for transfer performance compared to LSTM baselines?
• RQ4 Does language pretraining improve compute efficiency over random initialization when transferring to other modalities?
• RQ5 Which components (input layer, output layer, layer norm, positional embeddings) are most crucial to finetune?

Key findings

• Frozen pretrained transformers achieve competitive accuracy on seven downstream tasks compared to fully trained transformers and LSTMs.
• Bit Memory and XOR tasks reach 100% with FPT, while ListOps, MNIST, CIFAR-10, CIFAR-10 LRA, and Homology show substantial gains over LSTMs and approach full-transformer baselines.
• Language pretraining provides faster convergence than random initialization across tasks.
• Model performance scales with size; larger frozen transformers improve CIFAR-10 accuracy (68.2% base, 68.2% → 72.1% on CIFAR-10 with base setup, higher with larger variants).
• Freezing attention layers yields interpretable, task-relevant attention patterns on some bit tasks, indicating semantic alignment with downstream needs.