[Paper Review] TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

Augustus Odena, Ian Goodfellow|arXiv (Cornell University)|Jul 28, 2018

Adversarial Robustness in Machine Learning26 references138 citations

TL;DR

TensorFuzz introduces coverage-guided fuzzing for neural networks to discover rare-input failures, using approximate nearest neighbors to define coverage, and releases an open-source TensorFuzz library.

ABSTRACT

Machine learning models are notoriously difficult to interpret and debug. This is particularly true of neural networks. In this work, we introduce automated software testing techniques for neural networks that are well-suited to discovering errors which occur only for rare inputs. Specifically, we develop coverage-guided fuzzing (CGF) methods for neural networks. In CGF, random mutations of inputs to a neural network are guided by a coverage metric toward the goal of satisfying user-specified constraints. We describe how fast approximate nearest neighbor algorithms can provide this coverage metric. We then discuss the application of CGF to the following goals: finding numerical errors in trained neural networks, generating disagreements between neural networks and quantized versions of those networks, and surfacing undesirable behavior in character level language models. Finally, we release an open source library called TensorFuzz that implements the described techniques.

Motivation & Objective

• Motivate the need for reliable debugging tools for neural networks in critical applications.
• Introduce a coverage-guided fuzzing framework tailored to neural networks.
• Define a practical coverage metric using activation space proximity via approximate nearest neighbors.
• Demonstrate TensorFuzz’s ability to surface numerical errors, quantization disagreements, and undesirable language model behaviors.
• Release an open-source TensorFuzz library for broader adoption.

Proposed method

• Adapt coverage-guided fuzzing from traditional software to TensorFlow graphs by measuring coverage in activation space rather than code branches.
• Use a seed corpus of valid neural network inputs and mutate inputs to explore new activations.
• Compute coverage via approximate nearest neighbors to detect new activation states.
• Employ an objective function to identify inputs that cause desired/undesired model behaviors.
• Batch mutations to leverage hardware parallelism and address nondeterminism in computation graphs.

Experimental results

Research questions

• RQ1 Can coverage-guided fuzzing uncover numerical errors (e.g., NaNs) in trained neural networks more efficiently than random search?
• RQ2 Can CGF reveal disagreements between a neural network and its quantized version that single-point evaluations miss?
• RQ3 Can CGF surface undesirable behaviors in character-level language models not easily found by random testing?
• RQ4 Is an activation-space coverage metric based on approximate nearest neighbors practical and effective across neural network architectures?

Key findings

• CGF rapidly finds numerical errors like NaNs in trained networks whereas random search often misses them.
• CGF can generate disagreements between 32-bit and 16-bit quantized models within a localized input region (e.g., around seed images).
• CGF surfaces undesirable behavior in a character-level language model, producing more blacklist words than random search in a 24-hour run.
• Activation-space coverage using approximate nearest neighbors provides a practical, architecture-agnostic coverage signal for neural networks.