[Paper Review] Cramming: Training a Language Model on a Single GPU in One Day

Jonas Geiping, Tom Goldstein|arXiv (Cornell University)|Dec 28, 2022

Topic Modeling30 citations

TL;DR

The paper explores training a transformer-based language model completely from scratch on a single consumer GPU for 24 hours and assesses downstream GLUE performance, achieving BERT-like results with a carefully designed pipeline and data curation.

ABSTRACT

Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment where training language models is out of reach for most researchers and practitioners. While most in the community are asking how to push the limits of extreme computation, we ask the opposite question: How far can we get with a single GPU in just one day? We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. Aside from re-analyzing nearly all components of the pretraining pipeline for this scenario and providing a modified pipeline with performance close to BERT, we investigate why scaling down is hard, and which modifications actually improve performance in this scenario. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings. Through the lens of scaling laws, we categorize a range of recent improvements to training and architecture and discuss their merit and practical applicability (or lack thereof) for the limited compute setting.

Motivation & Objective

• Investigate how far a transformer language model can be trained from scratch on a single GPU within 24 hours.
• Re-examine and optimize components of the pretraining pipeline under severe compute constraints.
• Assess downstream GLUE performance to benchmark cramming against BERT and related baselines.
• Analyze how data choice and curation influence performance in a low-resource regime.

Proposed method

• Implement a PyTorch-based pretraining pipeline with automated mixed precision and restrict to standard components unless justified by compute constraints.
• Use a 128-token packed sequence setup with a WordPiece vocabulary of 32768 tokens and a lowercase, ASCII-only corpus derived from Wikipedia and Books Corpus.
• Systematically ablate architectural choices (e.g., attention biases, embedding schemes, normalization) and training hyperparameters to identify what yields gains under 24-hour budgets.
• Leverage scaling laws to interpret results and guide where gains are possible, focusing on data throughput and gradient computation efficiency rather than large architectural changes.
• Experiment with data sourcing and processing (The Pile, C4 subsets) and data filtering (compression-based filtering, sorting) to improve downstream performance.

Experimental results

Research questions

• RQ1 What downstream GLUE performance can be achieved when pretraining a language model from scratch on a single GPU for one day?
• RQ2 Which architectural, training, and data choices yield meaningful gains under extreme compute down-scaling?
• RQ3 How do scaling laws observed in large-scale training manifest in the cramming (low-resource) regime?
• RQ4 To what extent do data curation and dataset preprocessing impact downstream performance in a 24-hour pretraining scenario?

Key findings

• Performance in the cramming setting follows scaling laws similar to large-compute regimes, with larger models learning more per-gradient and smaller models offering throughput benefits.
• Major architectural redesigns provide limited gains under the 24-hour budget, while targeted optimizations that speed up gradient computation offer improvements without increasing model size.
• Careful data curation and processing (e.g., filtering uncompressible content, sorting sequences, increasing end-batch size) can improve downstream performance, particularly with data sources like C4 and The Pile.
• Skipping dropout during pretraining maximizes parameter updates within the single-epoch budget, with dropout retained during supervised fine-tuning to regularize.
• Compared to Izsak et al. (2021), the cramming recipe delivers substantial gains on GLUE tasks, achieving competitive MNLI, QQP, QNLI, and SST-2 results across different GPUs (RTX 2080 Ti, A4000, A6000).