[Paper Review] Autoformalization with Large Language Models

Yuhuai Wu, Albert Q. Jiang+5 more|arXiv (Cornell University)|May 25, 2022

Mathematics, Computing, and Information ProcessingComputer Science41 citations

TL;DR

Large language models can translate natural-language math into Isabelle/HOL with notable success (38/150 perfect cases, 25.3% overall), and autoformalized theorems used to boost a neural prover to 35.2% on MiniF2F, achieving new state-of-the-art results.

ABSTRACT

Autoformalization is the process of automatically translating from natural language mathematics to formal specifications and proofs. A successful autoformalization system could advance the fields of formal verification, program synthesis, and artificial intelligence. While the long-term goal of autoformalization seemed elusive for a long time, we show large language models provide new prospects towards this goal. We make the surprising observation that LLMs can correctly translate a significant portion ($25.3\%$) of mathematical competition problems perfectly to formal specifications in Isabelle/HOL. We demonstrate the usefulness of this process by improving a previously introduced neural theorem prover via training on these autoformalized theorems. Our methodology results in a new state-of-the-art result on the MiniF2F theorem proving benchmark, improving the proof rate from $29.6\%$ to $35.2\%$.

Motivation & Objective

• Demonstrate that LLMs can autoformalize natural-language mathematical statements into formal Isabelle/HOL code.
• Assess the quality of autoformalizations via human evaluation and BLEU scores on miniF2F-derived datasets.
• Show that autoformalized theorems can improve neural theorem provers via expert iteration.

Proposed method

• Use in-context learning with few-shot exemplars to prompt PaLM and Codex to translate natural-language statements into Isabelle code.
• Evaluate translations using BLEU against human-ground-truth formalizations on miniF2F-algebra and miniF2F-number_theory subsets.
• Perform human error analysis on 150 autoformalizations to identify failure modes.
• Apply an expert-iteration loop: generate proofs with a base prover, add successful proofs to training data, and fine-tune to obtain improved provers.

Experimental results

Research questions

• RQ1 Can large language models translate natural-language mathematical statements into Isabelle/HOL with high fidelity?
• RQ2 How do model scale and different models (PaLM variants, Codex) affect autoformalization quality?
• RQ3 Can autoformalized theorems improve neural theorem provers on standard benchmarks like miniF2F?
• RQ4 What are common failure modes in autoformalization and how might prompting or examples mitigate them?

Key findings

• Codex and large PaLM models can produce perfect Isabelle translations for a subset of cases (e.g., Case Study 1), and overall 25.3% of 150 evaluated autoformalizations are perfect.
• BLEU scores for autoformalizations improve with model scale: PaLM 8B ( algebra 31.49, number_theory 22.10 ), PaLM 64B ( algebra 43.13, number_theory 31.43 ), PaLM 540B ( algebra 50.30, number_theory 36.16 ), Codex ( algebra 57.13, number_theory 43.33 ).
• Using autoformalized theorems to train a neural theorem prover via expert iteration achieves state-of-the-art on miniF2F: base 29.9% on test, after 1 iteration 34.0%, after 2 iterations 35.2% on test.
• The two-iteration expert iteration with autoformalized data yields a 5.6 percentage-point improvement over the previous state-of-the-art.
• Case studies demonstrate both perfect translations and some failures (e.g., misalignment of informal definitions with Isabelle concepts) and show the impact of few-shot prompting.