[Paper Review] LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron, Thibaut Lavril+12 more|arXiv (Cornell University)|Feb 27, 2023

Natural Language Processing Techniques3,849 citations

TL;DR

LLaMA presents open foundation language models from 7B to 65B parameters trained on publicly available data, achieving competitive performance with larger closed models and enabling inference on modest hardware.

ABSTRACT

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Motivation & Objective

• Democratize access to LLM research by using publicly available data only.
• Demonstrate that smaller models can outperform larger ones under certain training/inference budgets.
• Show competitive performance of 13B–65B models against leading models on standard benchmarks.
• Highlight training/inference efficiency and practical deployment considerations for researchers.

Proposed method

• Utilize a transformer-based architecture with pre-normalization, SwiGLU activation, and RoPE rotary embeddings.
• Train multiple model sizes (7B, 13B, 33B, 65B) on ~1.0–1.4T tokens with a cosine learning rate schedule and AdamW optimizer.
• Use a memory-efficient causal attention implementation and checkpointing to speed training on large GPUs.
• Pre-train on a mixture of publicly available datasets (CommonCrawl, C4, GitHub, Wikipedia, Books, ArXiv, Stack Exchange) with careful deduplication and language filtering.
• Tokenize with BPE via SentencePiece, including digit-splitting and UTF-8 fallback for broad coverage.

Figure 1: Training loss over train tokens for the 7B, 13B, 33B, and 65 models. LLaMA-33B and LLaMA-65B were trained on 1.4T tokens. The smaller models were trained on 1.0T tokens. All models are trained with a batch size of 4M tokens.

Experimental results

Research questions

• RQ1 Can open, publicly sourced data suffice to train high-quality foundation models across multiple scales (7B–65B)?
• RQ2 How do smaller open models perform relative to larger proprietary models on standard benchmarks under varying inference budgets?
• RQ3 What architectural and training optimizations yield strong performance and practical inference efficiency for large language models?
• RQ4 To what extent can instruction-finetuning improve open models’ task performance without extensive custom data?
• RQ5 What are the biases, toxicity, and carbon-footprint implications of open foundation models trained on public data?

Key findings

• LLaMA-13B outperforms GPT-3 (175B) on most benchmarks despite being 10× smaller.
• LLaMA-65B is competitive with the top models (Chinchilla-70B and PaLM-540B) in several evaluations.
• LLaMA-65B shows strong performance in zero/few-shot settings on Natural Questions and TriviaQA, with 65B achieving state-of-the-art-like results on some tasks.
• Code-generation capabilities (HumanEval, MBPP) favor LLaMA over several non-code-tuned baselines, with larger gains at higher parameter counts.
• Instruction-finetuning (LLaMA-I) improves MMLU performance, reaching 68.9% for 65B, surpassing some other instruction-finetuned models of comparable size.
• The study documents biases and toxicity patterns (RealToxicityPrompts, CrowS-Pairs, WinoGender) and provides comparisons to GPT-3 and OPT, noting size-related trends and area-specific biases.
• A detailed carbon-footprint analysis estimates emissions for training different models in a common data-center context, highlighting substantial energy use and the authors’ intent for openness to reduce future emissions.

Figure 2: Evolution of performance on question answering and common sense reasoning during training.