[Paper Review] To Code, or Not To Code? Exploring Impact of Code in Pre-training

Viraat Aryabumi, Yixuan Su+7 more|arXiv (Cornell University)|Aug 20, 2024

Second Language Acquisition and LearningPsychology3 citations

TL;DR

This paper investigates the impact of including code in pre-training data for large language models, demonstrating that code significantly enhances performance across non-code tasks. Using controlled ablation studies on models from 470M to 2.8B parameters, the authors show that adding code boosts natural language reasoning by up to 8.2%, world knowledge by 4.2%, and code generation by 12x, with code quality and inclusion during cooldown further amplifying gains.

ABSTRACT

Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role in general LLMs' performance, there is only limited work analyzing the precise impact of code on non-code tasks. In this work, we systematically investigate the impact of code data on general performance. We ask "what is the impact of code data used in pre-training on a large variety of downstream tasks beyond code generation". We conduct extensive ablations and evaluate across a broad range of natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for models with sizes ranging from 470M to 2.8B parameters. Across settings, we find a consistent results that code is a critical building block for generalization far beyond coding tasks and improvements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. Our work suggests investments in code quality and preserving code during pre-training have positive impacts.

Motivation & Objective

• To systematically evaluate the impact of code data on non-code downstream tasks in large language models.
• To determine whether code data improves generalization beyond code-specific capabilities.
• To analyze how code quality, proportion, and training stage (including cooldown) affect model performance.
• To assess whether code inclusion yields measurable gains across diverse benchmarks, including reasoning, knowledge, and LLM-as-a-judge win-rates.

Proposed method

• Conducted large-scale, controlled pre-training experiments with models ranging from 470M to 2.8B parameters.
• Compared text-only pre-training against variants with increasing proportions of code data from diverse sources.
• Incorporated high-quality synthetic code and code-adjacent data (e.g., GitHub commits) to assess data quality impact.
• Performed ablation studies on code inclusion during both initial pre-training and subsequent cooldown phases.
• Used standardized evaluation across benchmarks: natural language reasoning (e.g., HellaSwag), world knowledge (e.g., MMLU), code generation (HumanEval), and LLM-as-a-judge win-rates.
• Applied quality filters to code data, removing low-quality or malformed entries (e.g., excessive numbers, 0x strings, top-level domain lists).

Experimental results

Research questions

• RQ1 How does including code in pre-training affect performance on non-code natural language tasks such as reasoning and world knowledge?
• RQ2 What is the impact of code data quality—specifically, synthetic vs. web-based code—on downstream model performance?
• RQ3 Does incorporating code during the cooldown phase of pre-training yield additional gains across multiple task types?
• RQ4 How do varying proportions of code in the pre-training mixture affect model generalization and performance?

Key findings

• The inclusion of code data in pre-training leads to a relative 8.2% improvement in natural language reasoning tasks compared to text-only pre-training.
• Code inclusion results in a 4.2% relative improvement in world knowledge benchmarks such as MMLU.
• Generative win-rates, as judged by LLM-as-a-judge, improved by 6.6% when code was included in pre-training.
• Code performance on HumanEval benchmarks improved by 12x when code was included in pre-training.
• Using high-quality synthetic code data led to a 9% relative increase in natural language reasoning and a 44% increase in code performance compared to web-based code.
• Including code during the cooldown phase further improved performance, yielding a 3.7% relative gain in NL reasoning, 6.8% in world knowledge, and 20% in code performance over non-code cooldown.