[Paper Review] A Linter for Isabelle: Implementation and Evaluation

Yecine Megdiche, Fabian Huch+1 more|arXiv (Cornell University)|Jul 21, 2022

Software Engineering Research4 citations

TL;DR

This paper presents a linter for Isabelle/HOL that automatically detects anti-patterns in formalizations, integrating with Isabelle/jEdit and offering a command-line tool. It identifies 59,573 issues in the Archive of Formal Proofs, with high-severity lints correlating strongly with maintainability problems, and shows through user studies that real-time linting reduces fix time to under a minute for 60% of users while improving code quality and satisfaction.

ABSTRACT

In interactive theorem proving, formalization quality is a key factor for maintainability and re-usability of developments and can also impact proof-checking performance. Commonly, anti-patterns that cause quality issues are known to experienced users. However, in many theorem prover systems, there are no automatic tools to check for their presence and make less experienced users aware of them. We attempt to fill this gap in the Isabelle environment by developing a linter as a publicly available add-on component. The linter offers basic configurability, extensibility, Isabelle/jEdit integration, and a standalone command-line tool. We uncovered 480 potential problems in Isabelle/HOL, 14016 in other formalizations of the Isabelle distribution, and an astonishing 59573 in the AFP. With a specific lint bundle for AFP submissions, we found that submission guidelines were violated in 1595 cases. We set out to alleviate problems in Isabelle/HOL and solved 168 of them so far; we found that high-severity lints corresponded to actual problems most of the time, individual users often made the same mistakes in many places, and that solving those problems retrospectively amounts to a substantial amount of work. In contrast, solving these problems interactively for new developments usually incurs only little overhead, as we found in a quantitative user survey with 22 participants (less than a minute for more than 60% of participants). We also found that a good explanation of problems is key to the users' ease of solving these problems (correlation coefficient 0.48), and their satisfaction with the end result (correlation coefficient 0.62).

Motivation & Objective

• To address the lack of automated tools for detecting low-quality formalization patterns in Isabelle, which hinder maintainability, reusability, and performance.
• To improve code quality in Isabelle/HOL by identifying and warning against common anti-patterns such as overuse of apply scripts and complex initial methods.
• To evaluate the linter’s impact on developer productivity, code quality, and user satisfaction through quantitative analysis and user studies.
• To enable integration into CI/CD pipelines and IDEs to support high-quality formalization standards, especially for large-scale projects like the AFP.

Proposed method

• The linter is implemented as a standalone add-on for Isabelle, with support for Isabelle/jEdit and a command-line interface.
• It performs static analysis on Isar theory files using 23 configurable checks, primarily derived from Gerwin’s Isabelle/HOL style guide.
• The tool analyzes outer syntax and structure to detect problematic patterns such as long apply scripts, unnecessary auto calls, and overly complex initial methods.
• Lint severity levels (info, warning, error) are assigned based on impact on maintainability and performance.
• The linter is evaluated on the Isabelle distribution, Isabelle/HOL, and the Archive of Formal Proofs (AFP), with manual fixes applied to 168 issues.
• A user study with 22 participants assessed time to fix lints, satisfaction, and understanding of lint explanations.

Experimental results

Research questions

• RQ1 RQ1: Do new users understand the underlying problems behind high-severity lints, even when clearly explained?
• RQ2 RQ2: How effective is the linter in motivating users to fix issues, and how does fix time differ between new and legacy code?
• RQ3 RQ3: Does the linter impact proof-checking performance, and does real-time feedback improve user productivity and code quality?
• RQ4 RQ4: What is the frequency and severity distribution of anti-patterns in large-scale Isabelle formalizations, such as the AFP and Isabelle distribution?

Key findings

• The linter detected 59,573 lint issues in the Archive of Formal Proofs, with 480 in Isabelle/HOL and 14,016 in other Isabelle distribution formalizations.
• High-severity lints—such as long apply scripts and complex initial methods—were found in 54% of the AFP’s lines of code, indicating widespread maintainability risks.
• In a user study, 60% of participants fixed lints in under a minute, and satisfaction with the final code correlated strongly with the clarity of lint explanations (r = 0.62).
• The average time to fix a lint in legacy code was an order of magnitude higher than in new developments, highlighting the value of proactive linting.
• The linter had no measurable performance impact on proof checking, and users reported improved code readability and maintainability after fixes.
• The tool successfully identified repetitive anti-patterns, such as unnecessary simplifier calls in structured proofs, which were often repeated by individual authors.