[Paper Review] Phi-4 Technical Report

Marah Abdin, Jyoti Aneja+25 more|arXiv (Cornell University)|Dec 12, 2024

Advanced Research in Science and Engineering24 citations

TL;DR

Phi-4 is a 14B parameter language model that leverages heavily synthetic data, advanced data curation, and novel post-training to surpass its GPT-4o teacher on STEM reasoning benchmarks while maintaining competitive performance at a small scale.

ABSTRACT

We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.

Motivation & Objective

• Motivate the need for high-quality synthetic data to improve reasoning and problem-solving in smaller LLMs.
• Propose a multi-stage data strategy combining synthetic data generation, organic data curation, and post-training refinements.
• Demonstrate that data quality and training regimen can outperform larger models on reasoning benchmarks.
• Show evidence that phi-4 matches or exceeds larger baselines on multiple reasoning-heavy tasks while maintaining efficiency.

Proposed method

• Develop a 14B decoder-only transformer with a 4096 context length (extended to 16K in midtraining) and multilingual tokenization.
• Generate ~400B unweighted tokens of synthetic data via diverse techniques (multi-agent prompting, self-revision, instruction reversal, pivotal token strategies).
• Curate and filter high-quality organic data (web, books, code) to seed synthetic generation and improve alignment with reasoning tasks.
• Apply a staged training regimen with pretraining, midtraining, and post-training; adjust data mixtures and curricula based on ablations and target benchmarks.
• Use post-training methods including refined SFT datasets and Direct Preference Optimization (DPO) with pivotal-token search to steer outputs toward high-accuracy reasoning and safety.

Figure 1 : Average performance of different models on the November 2024 AMC-10 and AMC-12 tests. This is the average score (with maximum score 150) over the four tests on 100 runs with temperature $t=0.5$ . We chose $t=0.5$ to follow simple-evals [ 24 ] . Error bars are $2\sigma$ of the estimate. On

Experimental results

Research questions

• RQ1 How does a data-centric training recipe with synthetic data influence reasoning capabilities in a 14B model?
• RQ2 Can synthetic data and refined post-training close the gap with much larger models on STEM benchmarks like GPQA and MATH?
• RQ3 What data mixtures and curricula maximize long-context reasoning and knowledge retention in phi-4?
• RQ4 What is the impact of post-training methods (SFT, DPO, pivotal token search) on safety, robustness, and alignment?

Key findings

• Phi-4 achieves strong performance on reasoning benchmarks relative to its size, including comparisons to larger models on several STEM tasks.
• Synthetic data is a central driver of performance, with targeted data generation and midtraining data mixtures improving long-context and reasoning capabilities.
• Post-training techniques (revised SFT, DPO with pivotal token search, judge-guided DPO) further refine outputs and enhance alignment and safety.
• A fresh, contamination-aware evaluation approach indicates phi-4’s gains are not driven by test-set leakage, including evaluation on fresh AMC-2024 contests showing competitive results on math benchmarks.
• Midtraining to extend context length (4K to 16K) combined with long-context data improves performance on long-context tasks.
• Data-ablation studies demonstrate that prioritizing synthetic data and targeted high-quality organic data yields better reasoning gains than uniform data mixes, with post-training balancing knowledge and reasoning strengths.

Figure 2 : 5-shot MMLU score for phase 2 pretraining runs with 4 and 12 epochs of synthetic data. All models are trained for the same token horizon, thus the model with 4 epochs of synthetic has seen more (unique) web tokens. We see that despite many epochs on synthetic data, we do not see overfitti