[Paper Review] Towards making formal methods normal: meeting developers where they are

Alastair Reid, Luke Church+4 more|arXiv (Cornell University)|Oct 30, 2020

Software Testing and Debugging Techniques28 references12 citations

TL;DR

This paper proposes integrating formal verification into mainstream software development by meeting developers where they are—using their existing workflows, testing practices, and artifacts like test harnesses. It advocates for a 'weekly cost-benefit' ratio model to drive adoption, focusing on incremental, practical benefits rather than full correctness proofs, with the goal of increasing formal method usage by two orders of magnitude within a decade.

ABSTRACT

Formal verification of software is a bit of a niche activity: it is only applied to the most safety-critical or security-critical software and it is typically only performed by specialized verification engineers. This paper considers whether it would be possible to increase adoption of formal methods by integrating formal methods with developers' existing practices and workflows. We do not believe that widespread adoption will follow from making the prevailing formal methods argument that correctness is more important than engineering teams realize. Instead, our focus is on what we would need to do to enable programmers to make effective use of formal verification tools and techniques. We do this by considering how we might make verification tooling that both serves developers' needs and fits into their existing development lifecycle. We propose a target of two orders of magnitude increase in adoption within a decade driven by ensuring a positive `weekly cost-benefit' ratio for developer time invested.

Motivation & Objective

• Address the low adoption of formal methods in mainstream software development, where less than 1% of developers currently use them.
• Overcome the barrier of high effort and disruption to existing development processes that currently limit formal methods to safety-critical systems.
• Shift focus from proving full correctness to delivering measurable, short-term benefits that justify developer time investment.
• Enable broader adoption by integrating formal verification tools with existing developer practices such as testing and unit testing.
• Establish a sustainable 'weekly cost-benefit' ratio where the time spent using formal tools is justified by immediate, observable gains.

Proposed method

• Leverage existing developer artifacts—especially test harnesses and unit tests—for formal verification by reusing them as specifications or inputs to verification tools.
• Draw parallels between property-based testing and formal verification to reduce the learning curve and increase tool familiarity.
• Design verification tooling that integrates seamlessly into existing CI/CD pipelines and IDEs, minimizing workflow disruption.
• Prioritize tool reuse and interoperability across verification tasks to avoid tool fragmentation and reduce maintenance overhead.
• Develop profiling and design-for-verification guidelines to improve predictability and performance when verifying complex code.
• Use ensembles of verification tools to mitigate the instability of results on undecidable or hard-to-prove properties, similar to handling flaky tests.

Experimental results

Research questions

• RQ1 How can formal verification be integrated into existing developer workflows without requiring a fundamental shift in development practices?
• RQ2 What would a 'weekly cost-benefit' ratio look like that justifies developer time spent on formal methods in everyday development?
• RQ3 To what extent can existing test artifacts be repurposed as formal specifications or verification inputs?
• RQ4 How can verification tools be made predictable and reliable enough to be trusted in routine development, especially when dealing with undecidable properties?
• RQ5 What design and tooling practices can reduce the cognitive and engineering burden of formal verification while still improving software trustworthiness?

Key findings

• The paper estimates that less than 1% of developers currently use formal verification, indicating a significant opportunity for broader adoption.
• A two orders of magnitude increase in adoption—equivalent to 100x more developers using formal methods—is proposed as a realistic and impactful goal within ten years.
• The weekly cost-benefit ratio is identified as a critical metric: developers will only adopt formal methods if the time invested yields visible, short-term benefits.
• Reusing test harnesses and unit tests as inputs to formal verification tools can significantly lower the barrier to entry and accelerate integration.
• The instability of verification results on complex or undecidable properties mirrors the problem of flaky tests, necessitating tooling support for profiling and predictability.
• Design-for-verification practices—modifying code to simplify verification—can improve tool performance and developer trust, similar to established hardware design principles.