[Paper Review] NaturalProofs: Mathematical Theorem Proving in Natural Language

Sean Welleck, Jiacheng Liu+4 more|arXiv (Cornell University)|Mar 24, 2021

Topic Modeling47 references12 citations

TL;DR

This paper introduces NaturalProofs, a multi-domain corpus of mathematical theorems and proofs written in natural mathematical language—combining symbolic notation and natural language—sourced from ProofWiki, the Stacks Project, and textbooks. It benchmarks neural sequence models on mathematical reference retrieval and generation tasks, showing that large-scale models outperform classical methods in-domain but still struggle with zero-shot generalization and full reference recovery.

ABSTRACT

Understanding and creating mathematics using natural mathematical language - the mixture of symbolic and natural language used by humans - is a challenging and important problem for driving progress in machine learning. As a step in this direction, we develop NaturalProofs, a multi-domain corpus of mathematical statements and their proofs, written in natural mathematical language. NaturalProofs unifies broad coverage, deep coverage, and low-resource mathematical sources, allowing for evaluating both in-distribution and zero-shot generalization. Using NaturalProofs, we benchmark strong neural methods on mathematical reference retrieval and generation tasks which test a system's ability to determine key results that appear in a proof. Large-scale sequence models show promise compared to classical information retrieval methods, yet their performance and out-of-domain generalization leave substantial room for improvement. NaturalProofs opens many avenues for research on challenging mathematical tasks.

Motivation & Objective

• To address the gap in machine learning research on informal mathematical reasoning by creating a unified corpus of mathematical statements and proofs in natural language.
• To enable evaluation of models on both in-distribution and zero-shot out-of-distribution generalization in mathematical reasoning.
• To develop and benchmark neural methods for mathematical reference retrieval and generation, emulating core aspects of human proof construction.
• To support progress in automated reasoning by bridging informal mathematical discourse with formal verification and educational applications.
• To provide a publicly available dataset and codebase to foster research in natural language-based mathematical reasoning and theorem proving.

Proposed method

• Construct a multi-domain corpus by unifying broad-coverage data from ProofWiki, deep-coverage data from the Stacks Project, and low-resource textbook data into a common schema.
• Represent mathematical statements and proofs using a taxonomy of LaTeX environments mapped to data types (e.g., theorem, definition, proof), with consistent formatting and metadata.
• Design a mathematical reference retrieval task: given a theorem, retrieve the set of references (theorems, lemmas, definitions) cited in its proof.
• Extend the retrieval task to reference generation, requiring recovery of both the identity and order of references in a proof.
• Implement and evaluate two neural models: a pairwise scoring model using BERT encoders for theorem-reference matching, and a joint parameterization model for end-to-end reference generation.
• Use a contrastive training objective with negative reference sampling to improve retrieval performance, and apply fine-tuning on the full reference set during inference.

Experimental results

Research questions

• RQ1 Can neural sequence models effectively retrieve relevant references for a given mathematical theorem in natural language proofs?
• RQ2 How well do neural models generalize to novel theorems from out-of-distribution domains such as textbooks or the Stacks Project?
• RQ3 To what extent can neural models recover the correct order and number of references in a proof, beyond simple retrieval?
• RQ4 How do large-scale neural models compare to classical information retrieval methods in the context of informal mathematical reasoning?
• RQ5 What are the key limitations of current neural models in capturing symbolic and logical structure in mathematical proofs?

Key findings

• Large-scale neural sequence models outperform classical information retrieval methods on in-domain mathematical reference retrieval tasks.
• Neural models show promise in generating reference sequences, but struggle to fully recover the correct number and order of references in proofs.
• Out-of-distribution generalization remains a significant challenge, indicating limited robustness to domain shifts in mathematical reasoning.
• The use of joint parameterization improves retrieval performance by refining the top of the ranked list, suggesting benefits of end-to-end training.
• Despite strong in-domain performance, neural models still fall short in leveraging symbolic mathematical content effectively for reasoning.
• The NaturalProofs corpus enables evaluation of both retrieval and generation tasks across diverse mathematical domains, offering a foundation for future research in informal theorem proving.