[Paper Review] MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning

Ke Wang, Houxing Ren+8 more|arXiv (Cornell University)|Oct 5, 2023

Computational Physics and Python Applications4 citations

TL;DR

MathCoder introduces a framework to enhance open-source LLMs' mathematical reasoning by fine-tuning them on a novel dataset, MathCodeInstruct, which interleaves natural language reasoning, code generation, and execution results. The method achieves state-of-the-art performance on GSM8K (83.9%) and MATH (45.2%), outperforming both open-source models and even GPT-4 on the competition-level MATH benchmark.

ABSTRACT

The recently released GPT-4 Code Interpreter has demonstrated remarkable proficiency in solving challenging math problems, primarily attributed to its ability to seamlessly reason with natural language, generate code, execute code, and continue reasoning based on the execution output. In this paper, we present a method to fine-tune open-source language models, enabling them to use code for modeling and deriving math equations and, consequently, enhancing their mathematical reasoning abilities. We propose a method of generating novel and high-quality datasets with math problems and their code-based solutions, referred to as MathCodeInstruct. Each solution interleaves natural language, code, and execution results. We also introduce a customized supervised fine-tuning and inference approach. This approach yields the MathCoder models, a family of models capable of generating code-based solutions for solving challenging math problems. Impressively, the MathCoder models achieve state-of-the-art scores among open-source LLMs on the MATH (45.2%) and GSM8K (83.9%) datasets, substantially outperforming other open-source alternatives. Notably, the MathCoder model not only surpasses ChatGPT-3.5 and PaLM-2 on GSM8K and MATH but also outperforms GPT-4 on the competition-level MATH dataset. The dataset and models will be released at https://github.com/mathllm/MathCoder.

Motivation & Objective

• To close the performance gap between open-source and closed-source LLMs in mathematical reasoning.
• To enable open-source models to emulate the GPT-4 Code Interpreter’s ability to interleave reasoning, code, and execution.
• To develop a high-quality, instruction-following dataset that combines natural language, code, and execution outputs for math problem solving.
• To design a customized supervised fine-tuning and inference pipeline that supports seamless code integration during reasoning.
• To achieve state-of-the-art results on standard math benchmarks using only open-source models.

Proposed method

• Constructing the MathCodeInstruct dataset by generating GPT-4 Code Interpreter-style solutions for 80k math problems, including natural language, code, and execution results.
• Using a two-stage data collection process: first, generating solutions for existing GSM8K and MATH problems; second, applying problem interpolation to create intermediate-difficulty problems.
• Employing a specialized tokenization scheme with special tokens (<$|$ text $|$>, <$|$ code $|$>, <$|$ execution $|$>) to structure the model input for interleaved reasoning.
• Fine-tuning Llama-2 base models using supervised fine-tuning with the MathCodeInstruct dataset to enable code-based reasoning.
• Designing a training and inference pipeline that mimics the GPT-4 Code Interpreter’s behavior, allowing models to generate, execute, and reason over code outputs.
• Applying self-distillation to further improve model performance on the augmented dataset.

Experimental results

Research questions

• RQ1 Can open-source LLMs achieve performance comparable to GPT-4 Code Interpreter in mathematical reasoning through code-integrated instruction tuning?
• RQ2 How effective is a dataset that interleaves natural language, code, and execution results in improving mathematical reasoning?
• RQ3 Can problem interpolation effectively generate intermediate-difficulty math problems to improve model generalization?
• RQ4 Does a customized fine-tuning pipeline that supports code execution and reasoning improve performance on challenging math benchmarks?
• RQ5 Can open-source models surpass both existing open-source models and even GPT-4 on competition-level math problems?

Key findings

• MathCoder achieves a state-of-the-art score of 83.9% on the GSM8K benchmark, significantly outperforming other open-source LLMs.
• On the MATH benchmark, MathCoder attains a 45.2% accuracy, surpassing all prior open-source models and even GPT-4 on the competition-level subset.
• The model outperforms both ChatGPT-3.5 and PaLM-2 on GSM8K and MATH, demonstrating strong generalization and reasoning capability.
• The MathCodeInstruct dataset, with its interleaved structure of text, code, and execution, enables more effective supervision than purely natural language or code-only solutions.
• Self-distillation and problem interpolation significantly improve model generalization, especially on harder problems.
• The customized fine-tuning and inference pipeline successfully enables open-source models to emulate the GPT-4 Code Interpreter’s reasoning-with-code workflow.