[Paper Review] On Benchmarking the Capability of Symbolic Execution Tools with Logic Bombs

Hui Xu, Zirui Neil Zhao+2 more|arXiv (Cornell University)|Dec 5, 2017

Software Testing and Debugging Techniques37 references3 citations

TL;DR

This paper proposes a fine-grained, efficient benchmarking framework for symbolic execution tools using logic bombs—small, targeted programs that trigger only when specific challenges (e.g., floating-point arithmetic, symbolic memory) are correctly handled. The approach evaluates KLEE, Angr, and Triton across 12 challenges, revealing that Angr outperforms the others, solving 21 of 27 test cases, while Triton struggled with floating-point and complex external calls.

ABSTRACT

Symbolic execution now becomes an indispensable technique for software testing and program analysis. There are several symbolic execution tools available off-the-shelf, and we need a practical benchmark approach to learn their capabilities. Therefore, this paper introduces a novel approach to benchmark symbolic execution tools in a fine-grained and efficient manner. In particular, our approach evaluates the performance of such tools against the known challenges faced by general symbolic execution techniques, such as floating-point numbers and symbolic memories. To this end, we first survey related papers and systematize the challenges of symbolic execution. We extract 12 distinct challenges from the literature and categorize them into two categories: symbolic-reasoning challenges and path-explosion challenges. Then, we develop a dataset of logic bombs and a framework to benchmark symbolic execution tools automatically. For each challenge, our dataset contains several logic bombs, each of which is guarded by a specific challenging problem. If a symbolic execution tool can find test cases to trigger logic bombs, it indicates that the tool can handle the corresponding problems. We have conducted real-world experiments with three popular symbolic execution tools: KLEE, Angr, and Triton. Experimental results show that our approach can reveal their capabilities and limitations in handling particular issues accurately and efficiently. The benchmark process generally takes only dozens of minutes to evaluate a tool. We release our dataset on GitHub as open source, with an aim to better facilitate the community to conduct future work on benchmarking symbolic execution tools.

Motivation & Objective

• To address the lack of fine-grained, unbiased evaluation methods for symbolic execution tools that are not dependent on real-world program variability.
• To identify and systematize the key challenges that hinder symbolic execution, categorizing them into symbolic-reasoning and path-explosion challenges.
• To design a scalable, automated benchmarking framework using logic bombs that isolate and test specific capabilities of symbolic execution tools.
• To empirically evaluate leading symbolic execution tools (KLEE, Angr, Triton) on a standardized dataset of logic bombs to reveal their strengths and limitations.
• To release the benchmark dataset and framework as open source to support future tool evaluation and development in the community.

Proposed method

• Systematically surveying 12 distinct challenges in symbolic execution from the literature, categorized into symbolic-reasoning (e.g., floating-point, symbolic memory) and path-explosion (e.g., loops, external calls) challenges.
• Designing a dataset of minimal, self-contained logic bombs—each guarded by a specific challenge—so that successful execution triggers a logic bomb, indicating tool capability.
• Creating an automated benchmark framework that compiles logic bombs into binaries, runs symbolic execution tools on them in batch mode, and validates test case correctness.
• Using small, focused programs to minimize side effects and reduce symbolic execution time, ensuring efficiency and accuracy in benchmarking.
• Employing binary-level analysis for tools like Angr and Triton, while using source-level analysis for KLEE, to ensure fair comparison across tooling paradigms.
• Validating results by checking whether generated test cases trigger the logic bombs, thereby confirming that the underlying challenge was correctly resolved.

Experimental results

Research questions

• RQ1 How can symbolic execution tools be evaluated in a fine-grained, reproducible, and unbiased manner independent of real-world program characteristics?
• RQ2 Which specific challenges (e.g., floating-point arithmetic, symbolic memory) most critically limit the effectiveness of symbolic execution tools?
• RQ3 To what extent do leading symbolic execution tools—KLEE, Angr, and Triton—handle complex symbolic reasoning and path explosion challenges?
• RQ4 Can a logic bomb-based benchmarking approach accurately and efficiently reveal tool-specific strengths and weaknesses across diverse symbolic execution challenges?
• RQ5 What are the practical limitations of current symbolic execution tools when handling real-world edge cases such as nested arrays, function pointers, and external function calls?

Key findings

• Angr solved 21 out of 27 logic bomb cases, demonstrating superior handling of symbolic reasoning and path explosion challenges compared to KLEE and Triton.
• KLEE solved 9 cases, primarily excelling in array and symbolic jump handling when using STP-based solvers, but failed on C++-specific features like vectors and lists.
• Triton solved only 3 cases, primarily due to lack of support for floating-point operations and complex external function calls like `atof`.
• Angr successfully handled two of five floating-point cases, but failed on all requiring decimal input conversion via `atof`, indicating limitations in external function modeling.
• KLEE failed on two-leveled array challenges due to insufficient array reasoning, while Angr succeeded, highlighting the importance of solver-level array support.
• Binary-level symbolic execution tools (Angr, Triton) were more resilient to low-level challenges like `jmp` instructions and stack layout issues than source-level tools like KLEE.