[Paper Review] Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs

Albert Q. Jiang, Sean Welleck+7 more|arXiv (Cornell University)|Oct 21, 2022

Logic, programming, and type systems25 citations

TL;DR

The DSP method converts informal proofs into formal proof sketches to guide an automated prover, substantially increasing miniF2F proof success in Isabelle from baselines around 20% to roughly 39% (human proofs) or up to 38-39% with language-model proofs.

ABSTRACT

The formalization of existing mathematical proofs is a notoriously difficult process. Despite decades of research on automation and proof assistants, writing formal proofs remains arduous and only accessible to a few experts. While previous studies to automate formalization focused on powerful search algorithms, no attempts were made to take advantage of available informal proofs. In this work, we introduce Draft, Sketch, and Prove (DSP), a method that maps informal proofs to formal proof sketches, and uses the sketches to guide an automated prover by directing its search to easier sub-problems. We investigate two relevant setups where informal proofs are either written by humans or generated by a language model. Our experiments and ablation studies show that large language models are able to produce well-structured formal sketches that follow the same reasoning steps as the informal proofs. Guiding an automated prover with these sketches enhances its performance from 20.9% to 39.3% on a collection of mathematical competition problems.

Motivation & Objective

• Address the scarcity of formal proofs by leveraging abundant informal proofs.
• Translate informal proofs into formal proof sketches to guide automated proving.
• Evaluate DSP with human-written and language-model–generated proofs on miniF2F in Isabelle.

Proposed method

• Introduce DSP consisting of three stages: drafting informal proofs, mapping to formal proof sketches via few-shot autoformalization, and filling gaps with automated provers.
• Use large language models (up to 175B parameters) to generate formal sketches from informal proofs.
• Pair informal proofs (human or LM) with formal sketches and apply off-the-shelf provers to close open conjectures.
• Experiment with miniF2F problems in Isabelle, using Sledgehammer with heuristics as a baseline and compare to Thor-based baselines.
• Perform ablations to assess the impact of in-line comments, drafting informal proofs, and automated provers.
• Analyze model variants (Codex, Minerva at various scales) and human proofs to assess effectiveness and memorization concerns.

Experimental results

Research questions

• RQ1 Can informal proofs be effectively translated into formal proof sketches that guide automated proving?
• RQ2 What is the impact of using human vs. language-model informal proofs on formalization success in Isabelle?
• RQ3 How do ablations (in-line comments, drafting, automated provers) affect performance and scalability of DSP?

Key findings

• DSP increases the proportion of problems solved by an automated prover on miniF2F from 20.9% to 38.9–39.3% depending on proof source and setting.
• Human informal proofs achieve 39.3% on miniF2F test; language-model proofs reach up to 38.9% with 540B Minerva and 37.7% with 62B Minerva on test set.
• Minerva 62B and 540B prove nearly as well as humans in guiding the prover, outperforming smaller LM baselines and some prior neural approaches on Isabelle.
• Ablation studies show inline informal-discussion comments in prompts improve success (e.g., +4.9% validation, +2.8% test); removing informal drafts reduces performance; automated provers are crucial (loss of ~9–10% without them).
• DSP enables solving 200/488 problems in miniF2F using human proofs and around 199 problems with certain LM configurations.
• The approach demonstrates robustness to noisy informal proofs, with some LM proofs correcting or leveraging incorrect steps in informal reasoning.