[Paper Review] DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

DeepSeek-AI, Qihao Zhu+38 more|arXiv (Cornell University)|Jun 17, 2024

vaccines and immunoinformatics approaches48 citations

TL;DR

DeepSeek-Coder-V2 is an open-source Mixture-of-Experts code model (16B and 236B) that approaches or surpasses closed-source models on code and math tasks, expands language support to 338 languages, and increases context length to 128K tokens.

ABSTRACT

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-V2, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.

Motivation & Objective

• Bridge the gap between open-source code models and closed-source leaders (e.g., GPT-4-Turbo) in coding and math tasks.
• Scale language and context coverage to support diverse programming languages and long-range code understanding.
• Improve coding and mathematical reasoning through multi-source pre-training, long-context training, and RL-based alignment.
• Release open-source models under permissive licenses to enable research and unrestricted commercial use.

Proposed method

• Continued pre-training from a DeepSeek-V2 intermediate checkpoint using an additional 6 trillion tokens, totaling 10.2 trillion training tokens across code, math, and natural language domains.
• Code corpus expanded from 86 to 338 programming languages and increased code tokens to 1,170B, with 60% code, 10% math, and 30% natural language in the mix.
• Extended context length from 16K to 128K tokens using the Yarn framework, with staged long-context training.
• Two-stage alignment: supervised fine-tuning on an instruction dataset combining code, math, and general data, followed by reinforcement learning with Group Relative Policy Optimization (GRPO) using a reward model tuned on compiler/test-case feedback.
• For 16B, employ Fill-In-Middle (FIM) training (Prefix-Suffix-Middle structure) to enhance code completion; for 236B, rely on Next-Token-Prediction objective.
• Compare with open- and closed-source baselines across coding, math, and NL tasks to validate performance gains.

Figure 1: The Performance of DeepSeek-Coder-V2 on math and code benchmarks.

Experimental results

Research questions

• RQ1 Can an open-source Mixture-of-Experts code model reach parity with or surpass closed-source leaders on coding and mathematical benchmarks?
• RQ2 How does expanding programming language coverage and context length affect coding accuracy and reasoning capabilities?
• RQ3 What is the impact of data composition, long-context training, and RL-alignment strategies on open-source code model performance?
• RQ4 To what extent can open-source models maintain general language capabilities while specializing in code and math tasks?

Key findings

• DeepSeek-Coder-V2 (236B) attains performance on par with or superior to GPT-4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro on code and math benchmarks.
• Code benchmarks show 90.2% on HumanEval and 76.2% on MBPP for the 236B model (EvalPlus pipeline), with 43.4% on LiveCodeBench.
• On math tasks, DeepSeek-Coder-V2 reaches 75.7% on MATH, nearly matching GPT-4o, and outperforms several closed-source models on AIME 2024 and related datasets.
• The model supports 338 programming languages and a maximum context length of 128K tokens, up from 86 languages and 16K context in prior work.
• Open-source DeepSeek-Coder-V2-Instruct achieves a 75.3% average on HumanEval/MBPP combining languages, second only to GPT-4o in the benchmark set presented.
• Long-context and FIM-enabled variants (16B) demonstrate strong code completion and repair capabilities, with notable results in FIM tasks and repair benchmarks.

Figure 2: Evaluation results on the “Needle In A Haystack” (NIAH) tests. DeepSeek-Coder-V2 performs well across all context window lengths up to 128K.