[Paper Review] Magicoder: Empowering Code Generation with OSS-Instruct

Yuxiang Wei, Zhe Wang+3 more|arXiv (Cornell University)|Dec 4, 2023

Topic Modeling23 citations

TL;DR

Magicoder is a family of open-source code LLMs (7B) trained with OSS-Instruct (75k data) and Evol-Instruct to boost code generation across Python, multilingual, and data-science tasks, rivaling and sometimes surpassing larger models and ChatGPT on benchmarks.

ABSTRACT

We introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B parameters. Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs through the wealth of open-source references for the production of more realistic and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1 ). Overall, OSS-Instruct opens a new direction for crafting diverse synthetic instruction data for code using abundant open-source references.

Motivation & Objective

• Mitigate bias in synthetic instruction data for code generation by leveraging open-source code as instruction sources.
• Produce high-quality, diverse, and controllable instruction-tuning data from real open-source code snippets.
• Build and evaluate a series of open-source Code LLMs (Magicoder and Magicoder-S) that outperform baselines at similar or smaller scales.
• Demonstrate the effectiveness of OSS-Instruct in improving parsing across languages and domains and enable open research by releasing data and weights.

Proposed method

• Generate 75K synthetic coding problems and solutions using OSS-Instruct from seed open-source code snippets.
• Decontaminate OSS-Instruct data to remove overlaps with standard benchmarks.
• Finetune CodeLlama-Python-7B to create Magicoder-CL and further finetune with Evol-Instruct to create Magicoder-S-CL.
• Evaluate on HumanEval, MBPP, MultiPL-E, and DS-1000, using EvalPlus extensions (HumanEval+, MBPP+).
• Perform ablations on language distribution and compare OSS-Instruct to direct finetuning on comment-function pairs.
• Open-source model weights, training data, and code.

Figure 1: Overview of OSS-Instruct and the pass@1 results of different LLMs on HumanEval (+)

Experimental results

Research questions

• RQ1 Does OSS-Instruct reduce data bias and improve instruction-tuning quality for code generation?
• RQ2 How do Magicoder and Magicoder-S variants perform relative to state-of-the-art open-source models and ChatGPT across diverse benchmarks?
• RQ3 What is the impact of training data language distribution on Python and multilingual code generation performance?
• RQ4 Is OSS-Instruct more effective than direct finetuning on semantically related but noisier data like comment-function pairs?

Key findings

• Magicoder-CL improves over the base CodeLlama-Python-7B and outperforms other open-source models of similar or smaller size.
• Magicoder-S-CL further improves results and surpasses ChatGPT on HumanEval+ while matching or exceeding on HumanEval.
• Magicoder-CL and Magicoder-S-CL achieve state-of-the-art performance among models with 7B parameters and show strong multilingual and data-science task performance (DS-1000).
• Magicoder-DS and Magicoder-S-DS (based on DeepSeek-Coder) achieve high pass@1 scores, with Magicoder-S-DS achieving 76.8 on HumanEval and outperforming some larger baselines with fewer finetuning tokens.
• OSS-Instruct data exhibits diversity and lower similarity to HumanEval than Self-Instruct or related methods, indicating improvements are not due to leakage.
• Direct finetuning on comment-function pairs can degrade performance, whereas OSS-Instruct data yields substantial gains.

Figure 2: The detailed prompt design for OSS-Instruct