[Paper Review] Lemur: Integrating Large Language Models in Automated Program Verification

Haoze Wu, Clark Barrett+1 more|arXiv (Cornell University)|Oct 7, 2023

Software Testing and Debugging TechniquesComputer Science3 citations

TL;DR

Lemur proposes a formal framework that integrates large language models (LLMs) with automated theorem provers for program verification, using LLMs to generate high-level proof goals (e.g., invariants) and verifiers to validate them. The approach achieves state-of-the-art results on challenging benchmarks, solving 26 out of 50 hard SV-COMP programs with multiple loops—demonstrating the first LLM-powered verification system to scale beyond single-loop programs.

ABSTRACT

The demonstrated code-understanding capability of LLMs raises the question of whether they can be used for automated program verification, a task that demands high-level abstract reasoning about program properties that is challenging for verification tools. We propose a general methodology to combine the power of LLMs and automated reasoners for automated program verification. We formally describe this methodology as a set of transition rules and prove its soundness. We instantiate the calculus as a sound automated verification procedure and demonstrate practical improvements on a set of synthetic and competition benchmarks.

Motivation & Objective

• To address the challenge of automated program verification, which requires high-level abstract reasoning beyond current formal tools' capabilities.
• To formalize the interaction between LLMs and automated reasoners in verification, filling a gap in theoretical grounding for hybrid AI-formal methods.
• To build a practical, sound, and terminating verification system that leverages LLMs for invariant generation and automated tools for validation.
• To evaluate the framework on diverse benchmarks, including synthetic programs and real-world competition-level C programs with complex control flow.
• To explore the scalability and robustness of LLM-guided verification on programs with multiple loops and non-trivial data types.

Proposed method

• Lemur formalizes a proof system using derivation rules that combine LLM-suggested proof goals (e.g., invariants) with automated verification checks.
• It defines a sound implication relation: q ⇒ p if p holds as an invariant in a program modified by assumption q.
• The framework uses LLMs to propose sub-goals (invariants or assumptions), which are then validated by automated reasoners such as esbmc or UAutomizer.
• A key innovation is the use of assumptions to simulate conditional program behavior, enabling the system to explore complex control paths.
• The system employs optimization techniques, including iterative refinement and prompt engineering, to improve LLM proposal quality and reduce redundant calls.
• The implementation integrates GPT-4 for invariant generation and integrates with existing C verifiers to ensure soundness and termination.

Experimental results

Research questions

• RQ1 Can LLMs reliably generate high-level proof goals (e.g., invariants) that guide automated verifiers toward correct verification?
• RQ2 Is there a formal calculus that ensures the soundness of combining LLM-generated reasoning with automated verification?
• RQ3 Can such a hybrid approach solve complex verification benchmarks with multiple loops that current tools fail to verify?
• RQ4 How does the performance of LLM-guided verification compare to purely AI-based or purely formal methods on standard benchmarks?
• RQ5 What are the practical limitations of LLMs and automated verifiers in this hybrid setting, and how can they be mitigated?

Key findings

• Lemur solved 26 out of 50 hard SV-COMP benchmarks within a 20-minute timeout, while esbmc and UAutomizer solved none, demonstrating a significant performance leap.
• The system successfully verified programs with up to four loops, marking the first time an LLM-powered approach has scaled to such complex control structures.
• On average, lemur required 9.1 proposals per solved benchmark—higher than the 4.7 on simpler Code2Inv benchmarks—indicating increased reasoning complexity.
• LLMs generated insightful invariants not constrained by syntactic templates, such as x%4==0, even when the modulo operator was absent in the source code.
• In several cases, LLMs produced disjunctive invariants that precisely captured loop behavior, outperforming traditional predicate abstraction techniques.
• The framework’s soundness is formally proven, making it the first such hybrid verification system with a complete formal calculus and proof of correctness.