[Paper Review] Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, Louis Martin+66 more|arXiv (Cornell University)|Jul 18, 2023

Topic Modeling2,594 citations

TL;DR

The paper introduces Llama 2, a family of open pretrained and fine-tuned LLMs (7B to 70B) including Llama 2-Chat, trained with enhanced data, longer context, GQA, and RLHF for dialogue, showing strong open-model performance and competitive safety.

ABSTRACT

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

Motivation & Objective

• Demonstrate an open family of pretrained LLMs (Llama 2) with larger data, longer context, and scalable architecture.
• Showcase fine-tuning techniques (SFT, RLHF) to create Llama 2-Chat optimized for dialogue.
• Improve safety, reduce data contamination, and provide a responsible release strategy for community use.

Proposed method

• Pretraining on a new mix of publicly available data, 2 trillion tokens, 4k context length for larger models, and Grouped-Query Attention (GQA).
• Fine-tuning involved supervised fine-tuning (SFT) with high-quality instruction data and a careful prompt/response objective.
• Applied reinforcement learning with human feedback (RLHF) using separate Helpfulness and Safety reward models trained on large human preference datasets.
• Introduced Ghost Attention (GAtt) to improve multi-turn dialogue flow.
• Employed red-teaming and safety-focused data annotation to evaluate and iterate on safety.
• Compared Llama 2-Chat to open-source and some closed-source baselines across standard benchmarks and human evaluations.

Experimental results

Research questions

• RQ1 How do open LLMs like Llama 2 compare to closed-models on standard benchmarks and human evaluations of helpfulness and safety?
• RQ2 What pretraining data, model architecture decisions (e.g., context length, GQA), and fine-tuning protocols yield strong open LLMs for dialogue?
• RQ3 Can separate reward models for helpfulness and safety improve RLHF outcomes for chat models?
• RQ4 What safety measures and evaluation procedures are effective for responsible open release of LLMs?

Key findings

• Llama 2-Chat generally outperforms existing open-source chat models on benchmarks and is competitive with some closed-source models in human evaluations.
• Llama 2-70B achieves strong results on MMLU and BBH benchmarks, approaching but not surpassing some closed models on certain tasks.
• A dual reward modeling setup (Helpfulness RM and Safety RM) yields superior reward predictions and RLHF performance compared to baselines.
• The authors report a responsible release strategy, data contamination analyses, red-teaming results, and a public safety/alignment framework to enable community contributions.
• Open releases are framed to reduce barriers to reproduction and encourage safe deployment, with task-specific safety testing recommended for applications.