[Paper Review] HOL(y)Hammer: Online ATP Service for HOL Light

Cezary Kaliszyk, Josef Urban|arXiv (Cornell University)|Sep 19, 2013

Logic, programming, and type systems24 references4 citations

TL;DR

HOL(y)Hammer is an online AI/ATP service that enables automated theorem proving in HOL Light by combining premise selection, translation to ATP formats, and parallel execution of 7 AI/ATP combinations and 4 decision procedures. It achieves a 47% success rate in proving Flyspeck theorems in 30 seconds on a 14-CPU workstation, with the online version now running on a 48-CPU server and supporting local installation for private use.

ABSTRACT

HOL(y)Hammer is an online AI/ATP service for formal (computer-understandable) mathematics encoded in the HOL Light system. The service allows its users to upload and automatically process an arbitrary formal development (project) based on HOL Light, and to attack arbitrary conjectures that use the concepts defined in some of the uploaded projects. For that, the service uses several automated reasoning systems combined with several premise selection methods trained on all the project proofs. The projects that are readily available on the server for such query answering include the recent versions of the Flyspeck, Multivariate Analysis and Complex Analysis libraries. The service runs on a 48-CPU server, currently employing in parallel for each task 7 AI/ATP combinations and 4 decision procedures that contribute to its overall performance. The system is also available for local installation by interested users, who can customize it for their own proof development. An Emacs interface allowing parallel asynchronous queries to the service is also provided. The overall structure of the service is outlined, problems that arise and their solutions are discussed, and an initial account of using the system is given.

Motivation & Objective

• To provide an online, scalable, and user-friendly AI/ATP service for formal mathematics in the HOL Light system.
• To automate the proof of conjectures in HOL Light developments by leveraging premise selection and ATPs trained on existing proofs.
• To support both public online access and local installation for private, offline, or high-performance computing use.
• To enable integration with interactive theorem proving workflows via Emacs, web, and command-line interfaces.
• To support evolving libraries and future extensions, including version control and secure handling of private developments.

Proposed method

• The system translates HOL Light formalizations into ATP-compatible formats, including untyped first-order, polymorphic typed first-order, and typed higher-order logic.
• It employs multiple premise selection techniques trained on proofs from existing libraries, such as Flyspeck, Multivariate Analysis, and Complex Analysis.
• For each query, the service runs 7 distinct AI/ATP combinations and 4 decision procedures in parallel, maximizing coverage and performance.
• The service uses content encoding and checkpointing to efficiently reuse previously computed knowledge across project versions.
• It supports both online access via a web interface and local installation with customizable training on user-specific developments.
• The system leverages strategy scheduling and learning from prior successes/failures to improve ATP performance over time.

Experimental results

Research questions

• RQ1 Can a scalable, online AI/ATP service significantly reduce the need for manual lemma selection and proof construction in large HOL Light developments?
• RQ2 How effective is the combination of premise selection, translation to ATP formats, and parallel execution of multiple AI/ATP systems in proving formal theorems?
• RQ3 To what extent can AI/ATP methods trained on large corpora like Flyspeck generalize to new conjectures in related mathematical libraries?
• RQ4 What architectural and performance trade-offs arise when scaling an AI/ATP service to support multiple libraries and versions with low-latency response?
• RQ5 How can privacy and security be ensured when allowing users to upload and update private formal developments on a shared service?

Key findings

• The system achieves a 47% success rate in automatically proving Flyspeck theorems in 30 seconds on a 14-CPU workstation, using the strongest combination of AI/ATP methods.
• The online version runs on a 48-CPU server and performs up to 70 distinct AI/ATP attempts per query through parallel execution of 7 combinations and 4 decision procedures.
• The service supports both public online access and local installation, enabling offline use and private development with customizable training.
• Content encoding and checkpointing allow efficient reuse of previously computed proof knowledge, reducing redundant computation during project updates.
• The system demonstrates that strategy scheduling and learning from prior results significantly improve ATP performance, especially under higher time limits.
• The approach shows strong potential for integrating with informal mathematical texts, paving the way for deeper semantic parsing of scientific natural language in future systems.