[Paper Review] Gemma: Open Models Based on Gemini Research and Technology

Gemma Team, Thomas Mesnard+106 more|arXiv (Cornell University)|Mar 13, 2024

Multi-Agent Systems and Negotiation220 citations

TL;DR

Gemma provides two open LLMs (2B and 7B) built from Gemini tech, with pretrained and fine-tuned checkpoints, evaluated on broad benchmarks for language, reasoning, and safety, and released with open inference code and safety tooling.

ABSTRACT

This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.

Motivation & Objective

• Provide open, high-quality language models derived from Gemini research for research and development.
• Evaluate performance across language understanding, reasoning, coding, and safety benchmarks.
• Assess safety, memorization, and responsible deployment aspects to inform open-model practices.
• Offer pretrained and instruction-tuned checkpoints to enable broad research and experimentation.
• Promote responsible open-weight releases with supporting tooling and documentation.

Proposed method

• Transformer decoder architecture with context length 8192 tokens.
• Two model scales: 2B (2048-dim, 18 layers) and 7B (3072-dim, 28 layers).
• Improvements: Multi-Query Attention or multi-head attention, Rotary Position Embeddings (RoPE), GeGLU activations, RMSNorm.
• Training with up to 6T tokens on TPUv5e using Pathways-style sharding and 2D torus hardware; single-controller GPT-style orchestration (Jax, Pathways, GSPMD).
• Data filtering to reduce unsafe or low-quality content and to avoid leakage of evaluation sets; vocab size 256k; 2B trained on 2T tokens and 7B on 6T tokens.
• Two-stage fine-tuning: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from Human Feedback (RLHF); instruction-formatting tokens for dialogue control.

Figure 1: Language understanding and generation performance of Gemma 7B across different capabilities compared to similarly sized open models. We group together standard academic benchmark evaluations by capability and average the respective scores; see Table 6 for a detailed breakdown of performanc

Experimental results

Research questions

• RQ1 How do Gemma models perform relative to similarly sized open models on automated and human-evaluated benchmarks?
• RQ2 What is the safety and memorization profile of Gemma models, including risk of exposing or memorizing sensitive information?
• RQ3 What is the impact of instruction-tuning regimes (SFT, RLHF) on downstream performance and safety?
• RQ4 How do open weights contribute to research, safety research, and downstream innovation?
• RQ5 What deployment and governance mechanisms are effective for responsible open-model releases?

Key findings

• Gemma 7B achieves strong performance relative to similar open models, including surpassing several in the MMLU benchmark (64.3%) and MBPP (44.4%).
• In human evaluations against Mistral v0.2 7B Instruct, Gemma-7B IT shows a 51.7% positive win rate on instruction-following prompts and 58% win rate on safety prompts (with confidence intervals provided).
• Gemma models demonstrate strong mathematics and coding capabilities, outperforming many open competitors on GSM8K, MATH, and MBPP.
• Pretraining memorization is low and comparable to PaLM families, with no memorized sensitive data detected; approximate memorization is higher but still within expected ranges.
• Open releases include both pretrained and fine-tuned checkpoints, and an open-source inference/serving codebase to facilitate research and safety analysis.
• Gemma emphasizes responsible deployment with safety benchmarks, red-teaming, model cards, and a Generative AI Responsible Toolkit for developers.

Figure 2: Comparing average memorization rates across model families. We compare the Gemma pretrained models to PaLM and PaLM 2 models of comparable size and find similarly low rates of memorization.