[Paper Review] NEUZZ: Efficient Fuzzing with Neural Program Learning

Dongdong She, Kexin Pei+4 more|arXiv (Cornell University)|Jul 15, 2018

Software Testing and Debugging Techniques31 references25 citations

TL;DR

NEUZZ proposes a novel fuzzing framework that uses neural network-based surrogate models to smooth the discontinuous branching behavior of real-world programs, enabling gradient-guided input generation. By learning a differentiable approximation of program behavior, NEUZZ achieves 3X higher edge coverage and discovers 31 previously unknown bugs across 10 real-world programs, significantly outperforming 10 state-of-the-art graybox fuzzers.

ABSTRACT

Fuzzing has become the de facto standard technique for finding software vulnerabilities. However, even state-of-the-art fuzzers are not very efficient at finding hard-to-trigger software bugs. Most popular fuzzers use evolutionary guidance to generate inputs that can trigger different bugs. Such evolutionary algorithms, while fast and simple to implement, often get stuck in fruitless sequences of random mutations. Gradient-guided optimization presents a promising alternative to evolutionary guidance. Gradient-guided techniques have been shown to significantly outperform evolutionary algorithms at solving high-dimensional structured optimization problems in domains like machine learning by efficiently utilizing gradients or higher-order derivatives of the underlying function. However, gradient-guided approaches are not directly applicable to fuzzing as real-world program behaviors contain many discontinuities, plateaus, and ridges where the gradient-based methods often get stuck. We observe that this problem can be addressed by creating a smooth surrogate function approximating the discrete branching behavior of target program. In this paper, we propose a novel program smoothing technique using surrogate neural network models that can incrementally learn smooth approximations of a complex, real-world program's branching behaviors. We further demonstrate that such neural network models can be used together with gradient-guided input generation schemes to significantly improve the fuzzing efficiency. Our extensive evaluations demonstrate that NEUZZ significantly outperforms 10 state-of-the-art graybox fuzzers on 10 real-world programs both at finding new bugs and achieving higher edge coverage. NEUZZ found 31 unknown bugs that other fuzzers failed to find in 10 real world programs and achieved 3X more edge coverage than all of the tested graybox fuzzers for 24 hours running.

Motivation & Objective

• To address the inefficiency of evolutionary fuzzing in discovering hard-to-trigger software vulnerabilities.
• To overcome the limitations of gradient-guided optimization in programs with discontinuous, non-differentiable behaviors.
• To develop a smooth surrogate function that approximates the discrete branching behavior of real programs.
• To integrate neural network-based surrogate models with gradient-guided input generation for improved fuzzing efficiency.
• To evaluate NEUZZ against state-of-the-art graybox fuzzers on real-world programs in terms of edge coverage and novel bug discovery.

Proposed method

• Train a neural network to learn a smooth, differentiable approximation of the target program’s branching behavior from execution traces.
• Use the trained neural surrogate to estimate gradients of the program’s behavior with respect to input mutations.
• Apply gradient-guided optimization to generate inputs that maximize code coverage or trigger new branches.
• Incrementally refine the neural surrogate model using new execution feedback during fuzzing.
• Combine the neural surrogate with a graybox fuzzer to guide input generation while maintaining compatibility with real program execution.
• Use backpropagation through the neural model to efficiently compute gradients even in the presence of discontinuities in the original program.

Experimental results

Research questions

• RQ1 Can a neural network effectively approximate the discontinuous, non-differentiable branching behavior of real-world programs?
• RQ2 Can gradient-guided input generation using a learned surrogate model outperform evolutionary guidance in fuzzing?
• RQ3 To what extent can a neural surrogate improve edge coverage and novel bug discovery in real-world software?
• RQ4 How does NEUZZ compare to state-of-the-art graybox fuzzers in terms of efficiency and effectiveness?
• RQ5 Can the surrogate model be incrementally updated during fuzzing to maintain accuracy and guide exploration?

Key findings

• NEUZZ achieved 3X higher edge coverage than all tested graybox fuzzers after 24 hours of execution on real-world programs.
• NEUZZ discovered 31 previously unknown bugs in 10 real-world programs that were undetected by 10 state-of-the-art fuzzers.
• The neural surrogate model successfully approximated complex branching behavior, enabling effective gradient-based input generation despite program discontinuities.
• NEUZZ significantly outperformed all 10 evaluated state-of-the-art graybox fuzzers in both edge coverage and novel bug discovery.
• The integration of gradient guidance with neural surrogate learning led to faster convergence on new code paths compared to evolutionary methods.
• Incremental refinement of the neural model during fuzzing improved long-term exploration efficiency and coverage.