[Paper Review] An Empirical Study of Path Feasibility Queries

Asankhaya Sharma|arXiv (Cornell University)|Feb 20, 2013

Software Testing and Debugging Techniques12 references3 citations

TL;DR

This paper evaluates the performance of SMT solvers on path feasibility queries in symbolic execution, focusing on quantifier-free bit-vector and array formulas (QF_ABV). It finds STP significantly outperforms Z3 by an order of magnitude, and uses this insight to design a new LLVM-based analysis, Change Value Analysis, which reduces query-solving time by 48% on SIR benchmarks.

ABSTRACT

In this paper we present a comparative study of path feasibility queries generated during path exploration based software engineering methods. Symbolic execution based methods are gaining importance in different aspects of software engineering e.g. proving properties about programs, test case generation, comparing different executions of programs. These methods use SMT solvers to check the satisfiability of path feasibility queries written as a formula in the supported theories. We study the performance of solving such path feasibility queries using SMT solvers for real world programs. Our path condition formulas are generated in a theory of quantifier free bit vectors with arrays (QF_ABV). We show that among the different SMT solvers, STP is better than Z3 by an order of magnitude for such kind of queries. As an application we design a new program analysis (Change Value Analysis) based on our study which exploits undefined behaviors in programs. We have implemented our analysis in LLVM and tested it with the benchmark of SIR programs. It reduces the time taken for solving path feasibility queries by 48%. The study can serve as guidance to practitioners using path feasibility queries to create scalable software engineering methods based on symbolic execution.

Motivation & Objective

• To evaluate the performance of SMT solvers on path feasibility queries extracted from real-world programs during symbolic execution.
• To identify the most efficient SMT solver for queries in the theory of quantifier-free bit vectors with arrays (QF_ABV), a common formalism in symbolic execution.
• To develop a new program analysis, Change Value Analysis, that leverages solver performance insights to improve scalability in software engineering tasks.
• To provide empirical guidance for practitioners building scalable symbolic execution-based tools in software engineering.

Proposed method

• Generated path condition formulas from real-world programs using symbolic execution, expressed in the QF_ABV logic.
• Evaluated multiple SMT solvers—specifically STP and Z3—on these formulas to measure performance in solving path feasibility queries.
• Designed and implemented a new program analysis, Change Value Analysis, in the LLVM compiler infrastructure to detect undefined behaviors.
• Used the empirical solver performance data to guide optimization choices in the analysis, focusing on reducing time spent on SMT solving.
• Executed the analysis on the SIR benchmark suite to measure end-to-end performance improvements.
• Measured the reduction in time taken to solve path feasibility queries by comparing the new analysis with baseline approaches.

Experimental results

Research questions

• RQ1 Which SMT solver performs best on path feasibility queries expressed in the QF_ABV logic for real-world programs?
• RQ2 How does the performance of STP compare to Z3 in solving QF_ABV path conditions derived from real software?
• RQ3 Can insights from SMT solver benchmarking be used to design a more efficient program analysis for symbolic execution?
• RQ4 To what extent can the performance of path feasibility solving be improved by optimizing the analysis pipeline based on solver characteristics?

Key findings

• STP outperforms Z3 by an order of magnitude in solving path feasibility queries formulated in the QF_ABV logic.
• The proposed Change Value Analysis, implemented in LLVM, reduces the time required to solve path feasibility queries by 48% on the SIR benchmark suite.
• The empirical performance gap between STP and Z3 is consistent and significant across diverse real-world programs.
• The study demonstrates that solver choice has a major impact on the scalability of symbolic execution-based analyses.
• The results provide strong empirical evidence that solver selection is a critical factor in designing efficient symbolic execution tools.
• The performance improvement achieved by the new analysis is directly attributable to the use of STP and optimized query generation based on the empirical findings.