[Paper Review] HOList: An Environment for Machine Learning of Higher-Order Theorem Proving (extended version)

Kshitij Bansal, Sarah M. Loos+3 more|arXiv (Cornell University)|Apr 5, 2019

Logic, programming, and type systems13 citations

TL;DR

HOList is an open-source reinforcement learning environment built on HOL Light for training deep learning models in higher-order theorem proving. It includes a comprehensive benchmark derived from formalized mathematics, including the Kepler conjecture, and introduces DeepHOL, a deep reinforcement learning agent that achieves strong performance on this challenging reasoning task.

ABSTRACT

We present an environment, benchmark, and deep learning driven automated theorem prover for higher-order logic. Higher-order interactive theorem provers enable the formalization of arbitrary mathematical theories and thereby present an interesting, open-ended challenge for deep learning. We provide an open-source framework based on the HOL Light theorem prover that can be used as a reinforcement learning environment. HOL Light comes with a broad coverage of basic mathematical theorems on calculus and the formal proof of the Kepler conjecture, from which we derive a challenging benchmark for automated reasoning. We also present a deep reinforcement learning driven automated theorem prover, DeepHOL, with strong initial results on this benchmark.

Motivation & Objective

• To create a scalable, open-source environment for training machine learning models on higher-order logic theorem proving.
• To develop a comprehensive benchmark grounded in real formalized mathematics, including foundational theorems and the proof of the Kepler conjecture.
• To enable deep reinforcement learning agents to learn effective proof search strategies in complex, open-ended mathematical domains.
• To demonstrate the feasibility of combining deep learning with formal theorem proving in higher-order logic.

Proposed method

• The framework is built on the HOL Light theorem prover, providing a structured environment for reinforcement learning in formal mathematics.
• The benchmark is derived from existing formalizations in HOL Light, including core theorems in calculus and the proof of the Kepler conjecture.
• A deep reinforcement learning agent, DeepHOL, is trained to select proof steps using policy networks that predict actions in the proof search space.
• The environment supports state representation learning, action space definition, and reward shaping based on proof progress and correctness.
• Training uses imitation learning and reinforcement learning to improve proof search efficiency and success rates.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn to prove theorems in higher-order logic using a realistic, formalized mathematical benchmark?
• RQ2 How effective is a learned policy in navigating complex proof search spaces compared to baseline strategies?
• RQ3 To what extent can pre-trained models generalize across diverse mathematical domains within higher-order logic?
• RQ4 Can imitation learning combined with reinforcement learning improve sample efficiency in theorem proving?

Key findings

• DeepHOL achieves strong performance on the HOList benchmark, demonstrating the feasibility of deep reinforcement learning in higher-order theorem proving.
• The benchmark derived from HOL Light's formalization of calculus and the Kepler conjecture provides a realistic and challenging testbed for automated reasoning.
• The integration of imitation learning with reinforcement learning improves sample efficiency and convergence speed in proof search.
• The open-source framework enables reproducible research and future development of AI systems for formal mathematics.