[Paper Review] DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

Daya Guo, Qihao Zhu+11 more|arXiv (Cornell University)|Jan 25, 2024

Software Engineering Research104 citations

TL;DR

DeepSeek-Coder provides open-source code LLMs (1.3B–33B) trained on 2T tokens with repository-level data and Fill-In-Middle training, achieving state-of-the-art open-source results and competitive performance against closed models.

ABSTRACT

The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

Motivation & Objective

• Address the research gap caused by closed-source dominance in code intelligence by developing open-source code models.
• Train a suite of repository-aware, long-context code LLMs from scratch on a large, high-quality corpus.
• Evaluate how repository-level data construction and Fill-In-Middle training impact cross-file code generation and overall code proficiency.
• Provide instruction-tuned variants and comprehensive benchmarks to assess multilingual coding ability and real-world applicability.

Proposed method

• Construct a large-scale open-source code corpus (87 languages, 2T tokens) with repository-level organization.
• Apply dependency-based topological ordering to arrange training data by file dependencies.
• Perform repository-level near-deduplication to preserve repository structure while reducing repetition.
• Pretrain decoder-only Transformers (1.3B, 6.7B, 33B) with next-token prediction and Fill-In-Middle (FIM) objectives (PSM/ MSP variants) and 16K context window.
• Extend RoPE context and use RoPE-based long-context scaling for up to 64K tokens (empirically best around 16K).
• Train DeepSeek-Coder-Instruct via instruction tuning using Alpaca-style data and a separate evaluation protocol.

Experimental results

Research questions

• RQ1 Can repository-level data construction improve cross-file code generation compared to file-level data?
• RQ2 What is the impact of Fill-In-the-Middle training configurations (PSM vs MSP) on code generation and completion performance?
• RQ3 How do open-source DeepSeek-Coder models compare to closed-source models (Codex, GPT-3.5, GPT-4) on standard code benchmarks and real-world tasks?
• RQ4 Does instruction tuning (DeepSeek-Coder-Instruct) bridge the gap to closed models in code tasks across languages?
• RQ5 What is the effect of long-context adaptation on performance in repository-scale coding scenarios?

Key findings

• DeepSeek-Coder-Base 33B achieves state-of-the-art performance among open-source code models across multiple benchmarks.
• DeepSeek-Coder-Instruct 33B surpasses GPT-3.5 Turbo on many benchmarks, narrowing the gap to GPT-4 among open models.
• DeepSeek-Coder-Base 7B is competitive with models five times larger (e.g., CodeLlama-33B) in some settings.
• Repository-level data construction significantly boosts cross-file code generation capabilities.
• Fill-In-Middle training with 50% PSM rate offers a balance between code completion and FIM effectiveness.
• DeepSeek-Coder-Instruct models show strong LeetCode Contest performance, with 33B outperforming several open-source baselines and approaching GPT-4-level results in some tasks.
• On DS-1000 data science benchmarks, DeepSeek-Coder-33B achieves notable library-specific accuracy across Matplotlib, NumPy, Pandas, PyTorch, SciKit-Learn, and TensorFlow.