[Paper Review] Mistral 7B

Albert Q. Jiang, Alexandre Sablayrolles+16 more|arXiv (Cornell University)|Oct 10, 2023

Natural Language Processing TechniquesComputer Science261 citations

TL;DR

Mistral 7B is a 7B language model with grouped-query and sliding window attention that outperforms open/open-weights baselines in multiple benchmarks and includes an instruct-finetuned variant.

ABSTRACT

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.

Motivation & Objective

• Demonstrate that a small, efficiently engineered 7B model can surpass larger open models on a broad set of benchmarks.
• Introduce architectural innovations (grouped-query attention and sliding window attention) to improve inference speed and sequence handling.
• Provide an instruсt-tuned variant and show its competitiveness against larger chat models.
• Showcase practical deployment tools and guardrail/content moderation capabilities for real-world use.

Proposed method

• Adopt grouped-query attention (GQA) to accelerate inference and reduce memory during decoding.
• Use sliding window attention (SWA) to extend effective context length with lower cost.
• Implement a rolling buffer cache to cap memory usage during decoding.
• Pre-fill and chunk long prompts to manage attention and cache during generation.
• Fine-tune a version of the model on instruction datasets to create Mistral 7B – Instruct.
• Release reference implementation and integration with vLLM, Skypilot, and Hugging Face.

Figure 1: Sliding Window Attention. The number of operations in vanilla attention is quadratic in the sequence length, and the memory increases linearly with the number of tokens. At inference time, this incurs higher latency and smaller throughput due to reduced cache availability. To alleviate thi

Experimental results

Research questions

• RQ1 Can a 7B model beat larger open models (7B/13B/34B) on diverse benchmarks including reasoning, math, and code generation?
• RQ2 Does architectural innovation (GQA + SWA) provide practical speedups and memory savings without sacrificing performance?
• RQ3 What is the performance gap between a base 7B model and an instruction-finetuned variant in chat-like benchmarks?
• RQ4 How do guardrails and content moderation capabilities function when deployed with a lightweight model?
• RQ5 How does Mistral 7B compare to existing open models in chat and instruction-following settings?

Key findings

• Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks.
• It also surpasses Llama 1 34B in mathematics and code generation benchmarks.
• The Mistral 7B – Instruct chat model outperforms Llama 2 13B – Chat and closely approaches 13B chat performance.
• Efficient attention mechanisms (GQA and SWA) enable faster inference and longer effective context with reduced memory.
• Guardrails and system prompts can steer outputs, with the system prompt improving safety and maintaining utility.
• Self-reflection content moderation yields high precision (99.4%) and robust recall (95.6%).

Figure 2: Rolling buffer cache. The cache has a fixed size of $W=4$ . Keys and values for position $i$ are stored in position $i\bmod W$ of the cache. When the position $i$ is larger than $W$ , past values in the cache are overwritten. The hidden state corresponding to the latest generated tokens ar