[Paper Review] Some approximations in Model Checking and Testing

Marie-Claude Gaudel, Richard Lassaigne+2 more|arXiv (Cornell University)|Apr 16, 2013

Formal Methods in Verification96 references3 citations

TL;DR

This paper presents a unified framework for approximation techniques in model checking and testing, combining logic, statistics, and complexity theory to enable efficient, probabilistically guaranteed verification of system properties. It introduces methods like bounded and approximate model checking, probabilistic abstraction, and randomized approximation schemes that ensure polynomial-time or constant-time verification with bounded error, particularly for LTL formulas in the EPF fragment.

ABSTRACT

Model checking and testing are two areas with a similar goal: to verify that a system satisfies a property. They start with different hypothesis on the systems and develop many techniques with different notions of approximation, when an exact verification may be computationally too hard. We present some notions of approximation with their logic and statistics backgrounds, which yield several techniques for model checking and testing: Bounded Model Checking, Approximate Model Checking, Approximate Black-Box Checking, Approximate Model-based Testing and Approximate Probabilistic Model Checking. All these methods guarantee some quality and efficiency of the verification.

Motivation & Objective

• To address the computational intractability of exact model checking and testing by introducing systematic approximation techniques.
• To unify diverse approximation methods in model checking and testing under a common theoretical foundation rooted in logic and statistics.
• To provide formal guarantees on the quality and efficiency of approximations, especially for probabilistic and temporal logic properties.
• To explore the trade-offs between computational efficiency and approximation accuracy in verification tasks.
• To enable scalable verification of large or infinite-state systems through randomized and abstraction-based approximations.

Proposed method

• Uses bounded model checking (BMC) to restrict path exploration to a finite length, reducing the problem to SAT solving with polynomial complexity.
• Applies Monte Carlo methods and randomized approximation schemes (RAS) to estimate the probability of LTL formula satisfaction with multiplicative error bounds.
• Employs the generalized zero-one estimator theorem to derive an optimal approximation algorithm (OAA) that ensures (ε,δ)-approximation of expected values with high probability.
• Introduces property testing and statistical learning techniques to approximate model-property equivalence and input coverage with complexity independent of system size.
• Utilizes probabilistic abstraction and edit distance metrics to approximate model-checking results when exact computation is infeasible.
• Combines learning-based model reconstruction with model checking to enable approximate black-box checking of unknown systems.

Experimental results

Research questions

• RQ1 How can approximation techniques in model checking and testing be formally grounded in logic and statistics to ensure both efficiency and accuracy?
• RQ2 What are the theoretical limits and guarantees of randomized approximation schemes in verifying probabilistic systems with LTL properties?
• RQ3 In what ways can model abstraction and learning be used to approximate system behavior in black-box checking and model-based testing?
• RQ4 How can the complexity of verification be reduced while maintaining bounded error in the approximation of system properties?
• RQ5 Can multiple approximation techniques—such as bounded checking, probabilistic abstraction, and statistical learning—be combined to improve scalability and accuracy?

Key findings

• The optimal approximation algorithm (OAA) achieves an (ε,δ)-approximation of the probability of satisfying an LTL formula in the EPF fragment, with a number of experiments proportional to Γ/μ, where Γ depends on ε and δ.
• For systems with small satisfaction probabilities (μ → 0), the expected number of experiments may become exponential, indicating a limitation of the RAS approach in such cases.
• The fixed-point iteration of the OAA algorithm provides a randomized approximation scheme for the full LTL probability problem, ensuring convergence to an (ε,δ)-approximation.
• Approximate model checking using Monte Carlo methods achieves polynomial-time complexity in the size of the model, with error bounds dependent on the number of sampled paths.
• Property testing-based approaches in approximate model checking achieve complexity independent of the system size, depending only on the approximation parameter ε.
• Approximate black-box checking using learning techniques guarantees that the reconstructed model is ε-close to the true IUT after N samples, where N depends only on ε and δ, not on system size.