[Paper Review] DeepSeek-V3 Technical Report

DeepSeek-AI, Aixin Liu+198 more|arXiv (Cornell University)|Dec 27, 2024

Distributed and Parallel Computing Systems206 citations

TL;DR

DeepSeek-V3 is a 671B Mixture-of-Experts language model with 37B active per token, featuring Multi-head Latent Attention and auxiliary-loss-free load balancing, trained on 14.8T tokens with FP8; it achieves strong open-source performance and competitive closed-source parity.

ABSTRACT

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

Motivation & Objective

• Advance open-source LLM capabilities using a large Mixture-of-Experts architecture.
• Improve training efficiency and stability with FP8 mixed precision and dual-pipe pipeline parallelism.
• Enhance inference efficiency via Multi-head Latent Attention and Cross-Node communication optimizations.
• Introduce an auxiliary-loss-free load balancing strategy and a Multi-Token Prediction objective to boost performance.
• Extend context length and perform post-training (SFT and RL) to align with human preferences.

Proposed method

• Employ Multi-head Latent Attention (MLA) for reduced KV cache during inference while preserving performance.
• Use a DeepSeekMoE architecture with an auxiliary-loss-free load balancing strategy to maintain balanced expert utilization.
• Introduce a Multi-Token Prediction (MTP) objective that densifies training signals and can aid speculative decoding.
• Implement FP8 mixed-precision training with tile-wise and block-wise quantization, and memory-saving techniques like recomputing RMSNorm and MLA up-projections.
• Develop DualPipe pipeline parallelism and optimized cross-node all-to-all kernels to hide communication overhead and enable fine-grained expert parallelism.
• Conduct cross-node communication strategy leveraging InfiniBand and NVLink to balance bandwidth and latency.

Experimental results

Research questions

• RQ1 What performance gains do MLA and DeepSeekMoE provide for inference and training efficiency at scale?
• RQ2 How does the auxiliary-loss-free load balancing strategy affect model performance and expert utilization compared to traditional auxiliary losses?
• RQ3 Does the Multi-Token Prediction objective improve training signal and downstream task performance?
• RQ4 What are the efficiency and stability implications of FP8 training and the DualPipe framework on a model of this scale?
• RQ5 How does DeepSeek-V3 perform on standard benchmarks (code, math, reasoning) relative to open-source and closed-source models?

Key findings

• DeepSeek-V3 Base outperforms other open-source bases in code and math benchmarks and approaches leading closed-source models on several tasks.
• It achieves 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA, with performance comparable to GPT-4o and Claude-Sonnet-3.5 on selected benchmarks.
• In factual knowledge, it surpasses open-source peers on SimpleQA and Chinese SimpleQA, particularly excelling in Chinese factual knowledge.
• On math benchmarks, it attains state-of-the-art results among non-long-CoT models and even surpasses some long-CoT baselines on specific tasks (e.g., MATH-500).
• For coding tasks, it is the top-performing model on LiveCodeBench, indicating strong coding capabilities; overall engineering benchmarks also show competitive performance against Claude-Sonnet-3.5.
• The training process is notably economical (2.788M GPU hours total) and highly stable without irrecoverable loss spikes or rollbacks.