[Paper Review] Systematic Evaluation and Usability Analysis of Formal Tools for Railway System Design

Alessio Ferrari, Franco Mazzanti+2 more|arXiv (Cornell University)|Jan 27, 2021

Model-Driven Software Engineering Techniques105 references4 citations

TL;DR

This paper presents a systematic evaluation of 13 formal tools for railway system design and a usability study of 7 tools involving railway practitioners. It finds that while usability and maturity are not major barriers, poor support for process integration—such as traceability and toolchain interoperability—is the primary obstacle to adoption, offering a practical, evidence-based guide for industry and tool developers.

ABSTRACT

Formal methods and supporting tools have a long record of success in the development of safety-critical systems. However, no single tool has emerged as the dominant solution for system design. Each tool differs from the others in terms of the modeling language used, its verification capabilities and other complementary features, and each development context has peculiar needs that require different tools. This is particularly problematic for the railway industry, in which formal methods are highly recommended by the norms, but no actual guidance is provided for the selection of tools. To guide companies in the selection of the most appropriate formal tools to adopt in their contexts, a clear assessment of the features of the currently available tools is required. To address this goal, this paper considers a set of 13 formal tools that have been used for railway system design, and it presents a systematic evaluation of such tools and a preliminary usability analysis of a subset of 7 tools, involving railway practitioners. The results are discussed considering the most desired aspects by industry and earlier related studies. While the focus is on the railway domain, the overall methodology can be applied to similar contexts. Our study thus contributes with a systematic evaluation of formal tools and it shows that despite the poor graphical interfaces, usability and maturity of the tools are not major problems, as claimed by contributions from the literature. Instead, support for process integration is the most relevant obstacle for the adoption of most of the tools.

Motivation & Objective

• To address the lack of guidance for railway companies in selecting formal tools for safety-critical system development.
• To conduct a systematic evaluation of 13 formal tools used in railway system design based on industry-relevant features.
• To perform a preliminary usability analysis of 7 tools with railway practitioners to assess real-world usability.
• To identify key barriers to adoption and provide actionable insights for practitioners, researchers, and tool developers.
• To share evaluation sheets and usability data to support replication, extension, and informed decision-making.

Proposed method

• Adapted the DESMET methodology for tool evaluation to define a comprehensive set of 27 evaluation features through expert brainstorming involving formal methods and railway domain experts.
• Conducted a three-assessor evaluation of 13 tools using documentation, live trials, and iterative triangulation to ensure consistency and reliability in feature assessment.
• Performed a usability study with railway practitioners via structured live demos and standardized questionnaires to assess perceived usability of 7 tools.
• Used a trade-off approach to balance expert insight with the time required to learn complex formal tools, ensuring practical relevance.
• Shared detailed evaluation sheets, feature definitions, and usability data openly to enable transparency, replication, and future extension.
• Aligned the evaluation framework with industry standards such as CENELEC EN 50128 and the needs identified in prior surveys of railway practitioners.

Experimental results

Research questions

• RQ1 What are the key features and capabilities of currently available formal tools for railway system design?
• RQ2 How usable are these tools from the perspective of railway practitioners, particularly in real-world development contexts?
• RQ3 What are the main barriers to the adoption of formal tools in the railway industry, according to practitioners and evaluators?
• RQ4 To what extent do usability and maturity of formal tools act as adoption barriers, based on empirical evidence?
• RQ5 How can the evaluation framework and data be used to guide tool selection and improvement in industrial settings?

Key findings

• The majority of the 13 evaluated formal tools are mature and well-established, with strong verification capabilities and industrial readiness.
• Despite limited graphical user interfaces, the usability of the tools—assessed via practitioner feedback—was rated as 'OK' to 'Good' on average, indicating usability is not a primary barrier to adoption.
• The most significant obstacle to adoption is poor support for process integration, particularly features like traceability, versioning, and integration with existing development workflows.
• The study found no strong evidence that the mathematical complexity of formal tools inherently limits their learnability, as practitioners reported manageable learning curves with proper training.
• Tool qualification and certification were not critical barriers, as railway systems are often certified without formal tools, and cost-benefit analyses of formal methods show tangible benefits in testing reduction.
• The shared evaluation data, including detailed tool sheets and usability results, provide a transparent, up-to-date resource for industry and research communities to inform tool selection and development.