[Paper Review] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Marah Abdin, Jyoti Aneja+127 more|arXiv (Cornell University)|Apr 22, 2024

Computational Physics and Python Applications147 citations

TL;DR

Phi-3-mini is a 3.8B parameter language model trained on 3.3T tokens that can run on a phone and matches or rivals larger models like Mixtral 8x7B and GPT-3.5, with further variants (7B and 14B) showing stronger performance; performance comes from a data-centric training recipe combining filtered web data and synthetic data, plus alignment for safety and chat.

ABSTRACT

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

Motivation & Objective

• Demonstrate that a small language model can achieve high performance through data quality and targeted post-training rather than simply scaling up model size.
• Show that phi-3-mini runs on-device (4K context; 128K long-context variant) and quantizes to 4-bit to fit ~1.8GB.
• Introduce phi-3-small and phi-3-medium as higher-capacity variants and compare their performance to larger baselines.
• Describe the dataset design, two-phase pretraining, post-training (SFT and DPO), and safety alignment processes.

Proposed method

• Use a transformer decoder with 3072 hidden dimension, 32 heads, 32 layers; 4K context (128K variant via LongRope).
• Train on 3.3T tokens using heavily filtered web data and synthetic data, in two phases (Phase-1 general knowledge, Phase-2 reasoning/niche skills).
• Post-train with supervised fine-tuning (SFT) and direct preference optimization (DPO) to align for safety, robustness, and chat-style use.
• Quantize phi-3-mini to 4-bit for ~1.8GB on-device deployment; evaluate on standard benchmarks with 5-shot or fewer prompts and a fixed evaluation pipeline.

Figure 1 : 4-bit quantized phi-3-mini running natively on an iPhone with A16 Bionic chip, generating over 12 tokens per second.

Experimental results

Research questions

• RQ1 Can a small language model (under 4B parameters) reach the performance of larger models when trained with a data-optimal regime and post-training alignment?
• RQ2 What are the trade-offs between model size, data quality, and context length for on-device LLM deployment?
• RQ3 How does data curation plus SFT and DPO affect safety, robustness, and instruction-following in small LLMs?
• RQ4 What is the impact of long-context extensions (128K) on answer quality compared to standard 4K contexts?

Key findings

• phi-3-mini (3.8B) achieves 69% on MMLU and 8.38 MT-bench, rivaling Mixtral 8x7B and GPT-3.5.
• phi-3-small (7B, preview) reaches 75% on MMLU and 8.7 MT-bench.
• phi-3-medium (14B, preview) reaches 78% on MMLU and 8.9 MT-bench.
• Long-context phi-3-mini-128K maintains quality similar to 4K while handling long-context tasks.
• 4-bit quantized phi-3-mini runs on iPhone 14 with A16 Bionic at 12+ tokens per second.
• SFT and DPO post-training reduce harmful responses and improve safety, robustness, and reasoning capabilities.
• A data-optimal regime is used to calibrate training data for small models, showing data quality can substitute for some scale in performance.

Figure 2 : Scaling law close to the “Data Optimal Regime” (from left to right: phi-1.5, phi-2, phi-3-mini, phi-3-small) versus Llama-2 family of models (7B, 13B, 34B, 70B) that were trained on the same fixed data. We plot the log of MMLU error versus the log of model size.