[Paper Review] Generating Natural Adversarial Examples

Zhengli Zhao, Dheeru Dua+1 more|arXiv (Cornell University)|Oct 31, 2017

Adversarial Robustness in Machine LearningComputer Science26 references364 citations

TL;DR

The paper introduces a framework to generate natural, legible adversarial examples by searching in a GAN latent space, enabling evaluation and interpretation of black-box classifiers across image and text domains.

ABSTRACT

Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.

Motivation & Objective

• Motivate robust evaluation of black-box classifiers by using natural, semantically meaningful adversaries.
• Propose a latent-space search method that yields legible adversaries that lie on the data manifold.
• Apply the approach to both image and text domains to reveal local decision behavior of models.
• Provide quantitative and human evaluations showing the usefulness of natural adversaries for robustness analysis.

Proposed method

• Train a Wasserstein GAN (WGAN) with an inverter to map data to latent z space and back to x space.
• Learn a latent-space perturbation procedure by perturbing the inverted latent z′ and decoding with the generator to obtain x̃.
• Define x* = Gθ(z*) where z* minimizes the distance to z′ subject to f(Gθ(z̃)) ≠ f(x).
• Use a loss L combining reconstruction error and divergence between z and Iγ(Gθ(z)) to enforce a normal latent distribution.
• Implement two search algorithms (iterative stochastic search and hybrid shrinking search) that are black-box friendly and guarantee adversaries.
• Adopt a coarse-to-fine search strategy to improve efficiency without gradient information.

Experimental results

Research questions

• RQ1 Can natural adversarial examples be generated for black-box classifiers by operating in a learned latent representation rather than the input space?
• RQ2 Do natural adversaries maintain grammatical/semantic validity across image and text domains while fooling models?
• RQ3 Does the robustness of classifiers, as measured by latent-space perturbations, correlate with traditional accuracy across tasks?
• RQ4 How do generated adversaries inform interpretation of model decisions in vision, textual entailment, and machine translation?

Key findings

• Generated adversaries are natural-looking and legible in images and text, providing meaningful insights into classifier decisions.
• More accurate classifiers tend to require larger latent-space perturbations (Δz) to change predictions, indicating robustness differences.
• The framework works for black-box models, including non-differentiable ones like Random Forests, across MNIST, LSUN, textual entailment, and translation tasks.
• Human evaluations confirm the naturalness and similarity of the adversaries to original inputs, supporting interpretability.
• Hybrid shrinking search achieves substantial speedups while maintaining effectiveness in finding adversaries.