[Paper Review] Model-checking Driven Black-box Testing Algorithms for Systems with Unspecified Components

Gaoyan Xie, Zhe Dang|arXiv (Cornell University)|Apr 19, 2004

Software Testing and Debugging Techniques33 references3 citations

TL;DR

This paper proposes a model-checking driven black-box testing approach that combines formal verification with testing to ensure correctness of systems with unspecified components. By generating test cases from model-checking conditions expressed as communication and witness graphs, the method enables sound, complete, and automatic verification of temporal properties (CTL/LTL) without requiring component source code.

ABSTRACT

Component-based software development has posed a serious challenge to system verification since externally-obtained components could be a new source of system failures. This issue can not be completely solved by either model-checking or traditional software testing techniques alone due to several reasons: 1) externally obtained components are usually unspecified/partially specified; 2)it is generally difficult to establish an adequacy criteria for testing a component; 3)components may be used to dynamically upgrade a system. This paper introduces a new approach (called {\em model-checking driven black-box testing}) that combines model-checking with traditional black-box software testing to tackle the problem in a complete, sound, and automatic way. The idea is to, with respect to some requirement (expressed in CTL or LTL) about the system, use model-checking techniques to derive a condition (expressed in communication graphs) for an unspecified component such that the system satisfies the requirement iff the condition is satisfied by the component, and which can be established by testing the component with test cases generated from the condition on-the-fly. In this paper, we present model-checking driven black-box testing algorithms to handle both CTL and LTL requirements. We also illustrate the idea through some examples.

Motivation & Objective

• Address the challenge of verifying system correctness when integrating externally obtained, unspecified software components.
• Overcome limitations of pure model-checking (requires component details) and pure black-box testing (inadequate coverage and no formal guarantees).
• Enable automatic, complete, and sound verification of temporal logic properties (CTL and LTL) in component-based systems.
• Provide a method to generate test cases from formal conditions that guarantee system-level property satisfaction if the component passes them.
• Support dynamic upgrades and integration in safety-critical systems where component behavior may vary across deployment environments.

Proposed method

• Use model-checking techniques to derive a formal condition (in the form of communication or witness graphs) that ensures the system satisfies a given CTL or LTL property.
• Represent the required behavior of the unspecified component as a finite-state graph structure (communication/witness graphs) derived from the system's specification.
• Apply bounded, nested depth-first search on these graphs to systematically generate test cases for black-box testing of the component.
• Ensure completeness by exploring all reachable paths in the graph representation under bounded depth, guaranteeing full coverage of the condition.
• Integrate test-case generation with on-the-fly black-box testing, allowing dynamic trimming of graphs and reuse of test results to improve efficiency.
• Generalize the approach to handle multiple components, asynchronous communication, and restricted classes of infinite-state systems (e.g., timed automata).

Experimental results

Research questions

• RQ1 How can model-checking be used to derive testable conditions for unspecified components in component-based systems?
• RQ2 Can a combination of formal methods and black-box testing achieve complete and sound verification of temporal logic properties without access to component source code?
• RQ3 What is the structure and generation process for communication and witness graphs that encode component behavior required for system-level correctness?
• RQ4 How can test-case generation be made efficient and scalable while maintaining full coverage of the derived formal conditions?
• RQ5 To what extent can the approach be extended to systems with multiple unspecified components or asynchronous interactions?

Key findings

• The proposed algorithms for both CTL and LTL are sound and complete, ensuring that system satisfaction of a temporal property is guaranteed iff the component satisfies the derived condition.
• For LTL, the maximum length of test-cases fed into the unspecified component is bounded by $ O(n \cdot m^2) $, where $ n $ is the host system’s state count and $ m $ is the component’s state count.
• For CTL, the test-case length is bounded by $ O(k \cdot n \cdot m^2) $, with $ k $ being the number of CTL operators in the formula.
• The method supports dynamic testing with on-the-fly graph trimming and result caching to reduce redundant test execution.
• The approach is generalizable to systems with multiple unspecified components and can be adapted to restricted infinite-state models such as timed automata.
• The framework enables verification of real-time systems with unspecified components by generating component test cases without requiring full system model-checking.