[Paper Review] Code Llama: Open Foundation Models for Code

Baptiste Rozière, Jonas Gehring+24 more|arXiv (Cornell University)|Aug 24, 2023

Model-Driven Software Engineering Techniques392 citations

TL;DR

Code Llama is a family of open-code foundation models (7B, 13B, 34B, 70B) derived from Llama 2, featuring code generation and infilling, long-context support, and instruction-following variants, achieving state-of-the-art open-model performance on multiple coding benchmarks.

ABSTRACT

We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.

Motivation & Objective

• Demonstrate that starting from Llama 2 and fine-tuning with code-heavy data yields superior open-code models.
• Introduce infilling capabilities and long-context fine-tuning to enable in-editor coding and repository-scale reasoning.
• Showcase language-specialized (Python) and instruction-following (Instruct) variants with safety and helpfulness improvements.
• Evaluate on standard code benchmarks (HumanEval, MBPP, APPS) and multilingual benchmarks (MultiPL-E).
• Provide models under a permissive license suitable for research and commercial use.

Proposed method

• Initialize from Llama 2 weights and train on a code-heavy dataset (~500B tokens; 70B uses ~1T).
• Apply a multitask objective combining autoregressive and infilling predictions for 7B, 13B, and 70B variants.
• Extend context length by LCFT, adjusting RoPE rotation frequencies to support up to 100,000-token inputs.
• Develop Code Llama - Python specialized models trained on Python-heavy data.
• Create Code Llama - Instruct by further fine-tuning with proprietary instruction data and a self-instruct generation pipeline (unit tests and solutions).
• Evaluate using zero-shot and few-shot prompts on HumanEval, MBPP, APPS, and multilingual MultiPL-E benchmarks.

Experimental results

Research questions

• RQ1 Can Code Llama outperform other open-code models on standard code benchmarks?
• RQ2 Does infilling training provide practical benefits with acceptable trade-offs in autocomplete quality?
• RQ3 How does long-context fine-tuning affect performance and extrapolate to 100k-token inputs?
• RQ4 What gains arise from Python specialization and instruction-following fine-tuning in terms of code generation quality, safety, and usefulness across languages?
• RQ5 How do the models perform in multilingual coding scenarios compared to other open models?

Key findings

• Code Llama variants achieve state-of-the-art performance among open models on several benchmarks (HumanEval, MBPP, MultiPL-E).
• Code Llama - Python 7B can outperform Llama 2 70B on HumanEval and MBPP in Python tasks.
• Infilling-enabled models reach strong infilling benchmarks and can incur only modest drops in autoregressive generation metrics.
• Long-context fine-tuning enables stable generation and extrapolation up to 100,000 tokens, with moderate impact on standard benchmarks.
• Code Llama - Instruct improves safety and helpfulness benchmarks with only modest code-generation cost.
• Across languages, Code Llama outperforms Llama 2 models of the same size, and Code Llama 7B competes with larger public models on multilingual tasks.