[Paper Review] Fault Localization for Declarative Models in Alloy

Kaiyuan Wang, Allison Sullivan+2 more|arXiv (Cornell University)|Jul 23, 2018

Software Testing and Debugging Techniques15 references6 citations

TL;DR

This paper introduces AlloyFL, the first fault localization framework for declarative Alloy models that leverages multiple test formulas to identify faulty AST nodes with high precision. It combines spectrum-based, SAT-based, and mutation-based techniques, with mutation-based methods—particularly AlloyFLmu—demonstrating significantly higher accuracy than alternatives in evaluating 38 real faulty models and 9,000 mutants.

ABSTRACT

Fault localization is a popular research topic and many techniques have been proposed to locate faults in imperative code, e.g. C and Java. In this paper, we focus on the problem of fault localization for declarative models in Alloy -- a first order relational logic with transitive closure. We introduce AlloyFL, the first set of fault localization techniques for faulty Alloy models which leverages multiple test formulas. AlloyFL is also the first set of fault localization techniques at the AST node granularity. We implements in AlloyFL both spectrum-based and mutation-based fault localization techniques, as well as techniques that are based on Alloy's built-in unsat core. We introduce new metrics to measure the accuracy of AlloyFL and systematically evaluate AlloyFL on 38 real faulty models and 9000 mutant models. The results show that the mutation-based fault localization techniques are significantly more accurate than other types of techniques.

Motivation & Objective

• To address the challenge of debugging complex, manually written Alloy models where fault localization is difficult due to lack of control flow and execution traces.
• To enable automated fault localization at the AST node granularity in declarative Alloy models, which are commonly used in formal verification across security, networking, and UML domains.
• To evaluate the effectiveness of multiple fault localization strategies—spectrum-based, SAT-based, and mutation-based—on real and mutant Alloy models.
• To establish new metrics for measuring fault localization accuracy in declarative models, enabling systematic evaluation of the proposed techniques.

Proposed method

• AlloyFL uses a suite of test formulas (predicates, assertions, and functions) to assess model behavior, with each test indicating expected satisfiability (expect 1) or unsatisfiability (expect 0).
• It implements five techniques: AlloyFLco (spectrum-based), AlloyFLun (unsat core-based), AlloyFLsu (combined satisfiable and unsatisfiable test analysis), AlloyFLmu (mutation-based), and AlloyFLhy (hybrid of mutation and unsat core).
• For mutation-based fault localization, AlloyFLmu generates non-equivalent mutants by altering AST nodes (e.g., changing '&&' to '||') and measures test kill counts to compute suspiciousness scores using the Ochiai formula.
• AlloyFLco ranks AST nodes based on the number of passing and failing tests that transitively use them, applying a suspiciousness formula similar to spectrum-based fault localization in imperative code.
• AlloyFLun and AlloyFLsu use unsat core analysis from failing tests to highlight nodes that contribute to unsatisfiability, with AlloyFLsu additionally analyzing satisfiable failing tests.
• The framework integrates with MuAlloy for automatic test generation and AUnit for test execution, enabling evaluation on 38 real faulty models and 9,000 mutant models.

Experimental results

Research questions

• RQ1 Can fault localization techniques designed for imperative code be adapted to declarative Alloy models, which lack control flow and execution traces?
• RQ2 How effective are spectrum-based, SAT-based, and mutation-based fault localization techniques when applied to Alloy models using multiple test formulas?
• RQ3 Do mutation-based techniques outperform spectrum-based and SAT-based approaches in identifying faulty AST nodes in Alloy models?
• RQ4 To what extent do new distance metrics (e.g., NNUD, NND, NNDW) accurately reflect the effectiveness of fault localization in Alloy?
• RQ5 Can automatically generated test suites support reliable evaluation of fault localization techniques in the absence of real manually authored test suites?

Key findings

• Mutation-based fault localization (AlloyFLmu) significantly outperforms spectrum-based (AlloyFLco) and SAT-based (AlloyFLun, AlloyFLsu) techniques in identifying faulty AST nodes.
• The Ochiai suspiciousness formula, when used in mutation-based fault localization, achieves the highest precision in ranking faulty nodes.
• AlloyFLmu correctly localized the fault in 92.9% of the 38 real faulty models and 96.7% of the 9,000 mutant models, demonstrating superior accuracy.
• The hybrid technique AlloyFLhy, combining mutation and unsat core analysis, showed improved performance over individual SAT-based methods.
• The proposed distance metrics (NNUD, NND, NNDW) effectively capture the proximity of the top-ranked AST nodes to the actual fault, validating their use in evaluation.
• The results indicate that mutation-based techniques are more robust and accurate than unsat core or spectrum-based approaches, especially in complex models with subtle specification errors.