[Paper Review] OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

Keiran Paster, Marco Dos Santos+2 more|arXiv (Cornell University)|Oct 10, 2023

Mathematics, Computing, and Information Processing4 citations

TL;DR

OpenWebMath is an open, 14.7B-token dataset of high-quality mathematical web text extracted from Common Crawl, preserving LaTeX notation and filtering for quality and relevance. Trained on this data, 1.4B parameter models outperform models trained on 20x more general-domain tokens, demonstrating that domain-specific pretraining significantly enhances mathematical reasoning in language models.

ABSTRACT

There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.

Motivation & Objective

• To address the lack of publicly available, high-quality web-based mathematical text datasets that preserve mathematical notation.
• To enable research in mathematical reasoning, memorization, and generalization by providing open access to a large-scale, curated dataset.
• To demonstrate that pretraining on domain-specific, high-quality mathematical content improves reasoning performance more effectively than general-domain pretraining.
• To release a reproducible pipeline for extracting, filtering, and deduplicating mathematical web content at scale.

Proposed method

• Extracted 14.7B tokens from Common Crawl using a pipeline that preserves LaTeX content and removes HTML boilerplate.
• Applied multi-stage filtering using LLMs and rule-based heuristics to retain only high-quality, English, mathematical content.
• Used a clustering and LLM-based classification approach to analyze document types and subject distribution across domains.
• Performed deduplication using embedding-based techniques to eliminate near-duplicate content.
• Trained 1.4B parameter language models on OpenWebMath and compared performance to models trained on The Pile and ProofPile.
• Evaluated models on MATH, GSM8k, and LILA-multiarith benchmarks to measure downstream reasoning performance.

Experimental results

Research questions

• RQ1 Can a large-scale, open dataset of web-based mathematical text with preserved LaTeX notation improve the reasoning capabilities of large language models?
• RQ2 How does pretraining on OpenWebMath compare to pretraining on 20x more general-domain text in terms of mathematical reasoning performance?
• RQ3 What is the impact of combining OpenWebMath with other mathematical datasets on model performance?
• RQ4 To what extent does the quality and domain specificity of pretraining data influence model generalization and memorization?
• RQ5 How effective is the proposed pipeline for extracting and filtering high-quality mathematical content from the open web?

Key findings

• Models trained on 14.7B tokens of OpenWebMath outperform Pythia 1.4B, which was trained on 300B tokens of The Pile, on MATH-Algebra-Easy and GSM8k benchmarks.
• The OpenWebMath-only model achieved lower perplexity on MATH and GSM8k than models trained on 20x more general-domain data.
• The 50/50 mixture of OpenWebMath and ProofPile yielded the best overall performance across all benchmarks tested.
• The dataset contains 14.7B tokens of high-quality mathematical content, with 88% of documents falling into mathematics, physics, computer science, or related technical domains.
• Forum posts (e.g., Math StackExchange) and educational content made up the largest categories, with 12% of documents not cleanly classifiable into core STEM subjects.
• The pipeline successfully preserved LaTeX notation and removed 90% of non-mathematical boilerplate, resulting in a clean, high-fidelity dataset suitable for pretraining.