[Paper Review] SAVIOR: Towards Bug-Driven Hybrid Testing

Yaohui Chen, Peng Li+6 more|arXiv (Cornell University)|Jun 18, 2019

Software Testing and Debugging Techniques51 references4 citations

TL;DR

SAVIOR is a novel hybrid software testing framework that shifts from coverage-driven to bug-driven testing by prioritizing concolic execution on branches with higher vulnerability potential and verifying all candidate vulnerabilities along execution paths using SMT constraints. It detects vulnerabilities 43.4% faster than Driller and 44.3% faster than QSYM, discovering 88 and 76 more security violations respectively in 24 hours on benchmark programs.

ABSTRACT

Hybrid testing combines fuzz testing and concolic execution. It leverages fuzz testing to test easy-to-reach code regions and uses concolic execution to explore code blocks guarded by complex branch conditions. However, its code coverage-centric design is inefficient in vulnerability detection. First, it blindly selects seeds for concolic execution and aims to explore new code continuously. However, as statistics show, a large portion of the explored code is often bug-free. Therefore, giving equal attention to every part of the code during hybrid testing is a non-optimal strategy. It slows down the detection of real vulnerabilities by over 43%. Second, classic hybrid testing quickly moves on after reaching a chunk of code, rather than examining the hidden defects inside. It may frequently miss subtle vulnerabilities despite that it has already explored the vulnerable code paths. We propose SAVIOR, a new hybrid testing framework pioneering a bug-driven principle. Unlike the existing hybrid testing tools, SAVIOR prioritizes the concolic execution of the seeds that are likely to uncover more vulnerabilities. Moreover, SAVIOR verifies all vulnerable program locations along the executing program path. By modeling faulty situations using SMT constraints, SAVIOR reasons the feasibility of vulnerabilities and generates concrete test cases as proofs. Our evaluation shows that the bug-driven approach outperforms mainstream automated testing techniques, including state-of-the-art hybrid testing systems driven by code coverage. On average, SAVIOR detects vulnerabilities 43.4% faster than DRILLER and 44.3% faster than QSYM, leading to the discovery of 88 and 76 more uniquebugs,respectively.Accordingtotheevaluationon11 well fuzzed benchmark programs, within the first 24 hours, SAVIOR triggers 481 UBSAN violations, among which 243 are real bugs.

Motivation & Objective

• To address the inefficiency of coverage-driven hybrid testing in detecting real vulnerabilities despite high code coverage.
• To reduce the time delay in discovering exploitable vulnerabilities by prioritizing seeds and paths with higher exploitability potential.
• To improve detection accuracy by verifying all vulnerable candidates along execution paths using formal proofs.
• To outperform existing hybrid testing tools in both speed and completeness of vulnerability discovery.

Proposed method

• SAVIOR employs a bug-driven prioritization strategy that ranks branch conditions based on their potential to expose vulnerabilities, guiding concolic execution toward high-risk paths.
• It uses SMT constraints to model and verify all potential vulnerabilities along the execution path during concolic execution, proving their feasibility or non-existence.
• The framework dynamically selects seeds for fuzz testing based on their likelihood to trigger exploitable defects, rather than solely on code coverage.
• It integrates vulnerability labeling and symbolic execution to inspect all candidate faults, enabling precise identification of real security violations.
• SAVIOR combines fuzzing and concolic execution in a feedback-driven loop, where verified vulnerabilities guide further exploration.
• It instruments programs with UBSan-style checks and vulnerability labels to enable fine-grained tracking of potential defects during execution.

Experimental results

Research questions

• RQ1 Can a bug-driven approach outperform coverage-driven hybrid testing in vulnerability detection efficiency?
• RQ2 How effective is SAVIOR in identifying exploitable vulnerabilities compared to state-of-the-art tools like Driller and QSYM?
• RQ3 To what extent does bug-guided verification reduce false positives and improve detection accuracy?
• RQ4 How does prioritizing high-potential branches impact the time-to-detection of real security violations?
• RQ5 Can formal verification via SMT constraints improve the reliability of vulnerability detection in hybrid testing?

Key findings

• SAVIOR detects vulnerabilities 43.4% faster on average than Driller and 44.3% faster than QSYM in the same 24-hour testing window.
• It discovers 88 more security violations than Driller and 76 more than QSYM within 24 hours on 11 well-known fuzzing benchmarks.
• The framework identifies 481 unique security violations in the first 24 hours, demonstrating superior coverage and detection speed.
• By verifying all candidate vulnerabilities along execution paths, SAVIOR reduces false positives and increases confidence in reported findings.
• The bug-guided verification technique successfully proves the feasibility or non-existence of vulnerabilities using SMT constraints, enhancing reliability.
• The evaluation confirms that prioritizing high-potential paths significantly reduces time-to-detection without sacrificing coverage.