[Paper Review] Formal Algorithms for Transformers

Mary Phuong, Marcus Hütter|arXiv (Cornell University)|Jul 19, 2022

Energy Load and Power Forecasting52 citations

TL;DR

This paper provides a self-contained, mathematically precise overview of transformer architectures and formal pseudocode for their core algorithms, training, and inference, without presenting empirical results.

ABSTRACT

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

Motivation & Objective

• Explain what transformers are and their core architectural components.
• Present a compact, formal pseudocode description of transformer algorithms (about 50 lines).
• Describe tokenization, training, inference, and practical considerations for transformers.
• Compare prominent transformer architectures (encoder-only, decoder-only, and encoder-decoder) and their use cases.
• Provide a reference-like foundation to implement transformers from scratch and to reason about their properties.

Proposed method

• Define a formal notation system for transformers and their training and inference procedures.
• Present token embedding and positional embedding algorithms.
• Detail the attention mechanism with single-head and multi-head variants, including masking, cross-attention, and head aggregation (Algorithms 3–5).
• Introduce layer normalization variants and the unembedding step (Algorithms 6–7).
• Describe end-to-end transformer architectures including encoder-decoder, BERT-style encoder, and GPT-style decoder (Algorithms 8–9).
• Provide a concise, formal description intended to facilitate implementation and theoretical analysis of transformer models.

Experimental results

Research questions

• RQ1 What is the minimal, formal set of algorithms needed to describe transformer architectures, training, and inference?
• RQ2 How can tokenization, attention, and normalization be formalized to enable precise implementation from scratch?
• RQ3 How do encoder-only, decoder-only, and encoder-decoder transformer variants differ in their masking and attention schemes?
• RQ4 Can a compact pseudocode–level specification capture the essential operations of modern transformers for theoretical and practical use?

Key findings

• The document offers an essentially complete pseudocode description of transformer components and workflows in under 50 lines.
• It covers tokenization, embedding, positional encoding, various attention mechanisms, and unembedding, with explicit algorithms for training and prompting/inference.
• It surveys prominent transformer architectures (EDTransformer, BERT, GPT) and links them to the corresponding algorithms and sections.
• It argues for the value of formal algorithms to improve precision, reproducibility, and notational conventions in DL publications.
• It aims to serve theoreticians and practitioners who need compact, precise Transformer specifications for implementation and analysis.