[Paper Review] A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Hugh Zhang, Jeff Da+13 more|arXiv (Cornell University)|May 1, 2024

Educational Assessment and Pedagogy10 citations

TL;DR

The paper introduces GSM1k, a new 1250-problem benchmark designed to mirror GSM8k to assess whether LLM performance on grade-school arithmetic reflects true reasoning or data contamination, finding substantial overfitting in several model families but strong generalization in frontier models.

ABSTRACT

Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes. Further analysis suggests a positive relationship (Spearman's r^2 = 0.36) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that some models may have partially memorized GSM8k. Nevertheless, many models, especially those on the frontier, show minimal signs of overfitting, and all models broadly demonstrate generalization to novel math problems guaranteed to not be in their training data.

Motivation & Objective

• Assess whether GSM8k-style benchmarks suffer from data contamination by constructing an independent, human-made GSM1k dataset of grade-school math problems.
• Compare GSM1k to GSM8k across multiple model families and sizes to quantify overfitting and generalization.

Proposed method

• Construct GSM1k with 1250 human-annotated problems matching GSM8k difficulty distribution.
• Evaluate open- and closed-source LLMs on GSM1k using a fork of EleutherAI LM Evaluation Harness.
• Analyze overfitting by comparing performance gaps GSM8k vs GSM1k across model families.
• Measure model likelihood of generating GSM8k examples to assess data contamination.
• Provide qualitative lessons and discuss potential sources of overfitting beyond contamination.

Figure 1 : Notable models arranged by their drop in performance between GSM8k and GSM1k (lower is worse). We notice that Mistral and Phi top the list of overfit models, with almost 10% drops on GSM1k compared to GSM8k, while models such as Gemini, GPT, and Claude show little to no signs of overfitti

Experimental results

Research questions

• RQ1 Does GSM1k reveal overfitting in LLMs that GSM8k may mask?
• RQ2 Which model families show systematic overfitting across sizes and releases?
• RQ3 Do frontier models exhibit less overfitting and better generalization to new problems?
• RQ4 What is the relationship between a model's likelihood of generating GSM8k data and its GSM8k–GSM1k performance gap?
• RQ5 To what extent can data contamination explain observed overfitting, and are there other contributing factors?

Key findings

• GSM1k shows accuracy drops of up to 13% on GSM1k versus GSM8k for several model families.
• Mistral and Phi families exhibit systematic overfitting across model sizes; frontier models show minimal overfitting.
• Frontier models (e.g., Gemini, GPT, Claude) perform similarly on GSM8k and GSM1k, suggesting stronger generalization or contamination protection.
• A positive relationship (Spearman r^2 = 0.32) exists between a model’s likelihood of generating GSM8k data and its GSM8k–GSM1k performance gap, indicating partial memorization of GSM8k test data.
• Overfit models can still reason effectively and solve novel GSM1k problems, challenging the notion that overfitting eliminates reasoning ability.
• Data contamination is likely not the sole explanation for overfitting; other factors such as benchmark-driven data collection may contribute.

Figure 3 : Approximate difficulty distribution of GSM8k train and test sets, measured by number of required steps to solve the problem. GSM1k annotators were instructed to create problems matching the overall distribution of the combined train and test difficulty distribution. The process of estimat