[Paper Review] Adversarial Manipulation of Deep Representations

Sara Sabour, Yanshuai Cao+2 more|arXiv (Cornell University)|Nov 16, 2015

Adversarial Robustness in Machine Learning13 references67 citations

TL;DR

This paper introduces 'feature adversaries'—adversarial images that are perceptually similar to a source image but have deep neural network (DNN) representations nearly identical to a different, target guide image. Using gradient-based optimization to minimize representation distance in intermediate DNN layers while constraining perceptual distortion, the method generates adversarial images with natural-looking internal features, revealing a fundamental vulnerability in DNN representations beyond misclassification.

ABSTRACT

We show that the representation of an image in a deep neural network (DNN) can be manipulated to mimic those of other natural images, with only minor, imperceptible perturbations to the original image. Previous methods for generating adversarial images focused on image perturbations designed to produce erroneous class labels, while we concentrate on the internal layers of DNN representations. In this way our new class of adversarial images differs qualitatively from others. While the adversary is perceptually similar to one image, its internal representation appears remarkably similar to a different image, one from a different class, bearing little if any apparent similarity to the input; they appear generic and consistent with the space of natural images. This phenomenon raises questions about DNN representations, as well as the properties of natural images themselves.

Motivation & Objective

• To investigate whether deep neural network (DNN) representations can be manipulated to mimic those of a different natural image while preserving perceptual similarity to the original image.
• To explore whether such adversarial images are generic and indistinguishable from natural image representations across multiple DNN layers.
• To determine whether the phenomenon arises from network architecture, training data, or inherent model properties.
• To contrast this new class of adversarial examples with prior work focused solely on misclassification.
• To assess the role of model linearity and generalization in enabling such representation-level manipulation.

Proposed method

• Formulate the adversarial image generation as a constrained optimization problem: minimize the L2 distance between the DNN representation of the perturbed image and the guide image’s representation at a chosen layer.
• Apply an L∞ norm constraint on pixel-wise perturbations (‖I − Is‖∞ < δ) to ensure imperceptibility to human observers.
• Use gradient-based optimization to solve the constrained minimization problem, iteratively updating the image to reduce representation distance to the guide.
• Introduce a linear approximation baseline (feature-linear) using the Jacobian of the DNN layer to test the linearity hypothesis of representation shifts.
• Evaluate the method on a trained CaffeNet model and compare results with randomly initialized networks to isolate architectural effects.
• Analyze the sparsity and density of adversarial representations in feature space to assess their naturalness and genericity.

Experimental results

Research questions

• RQ1 Can deep neural network representations be manipulated to match those of a different natural image while preserving perceptual similarity to the source image?
• RQ2 Are the resulting adversarial images indistinguishable from natural images in terms of their internal DNN representations across multiple layers?
• RQ3 Does the existence of such feature adversaries depend on the training data or is it inherent to the network architecture?
• RQ4 To what extent does the linearity of DNN representations explain the success of this adversarial manipulation?
• RQ5 How do adversarial representations compare in distribution and density to natural image representations in the DNN feature space?

Key findings

• The proposed method successfully generates adversarial images that are perceptually similar to the source image but have DNN representations that are 50% or less distant from the guide image’s representation at layers C2 and deeper.
• Feature adversaries achieve significantly lower representation distances than the linear approximation baseline (feature-linear), which fails to reduce distance below 80% of the original source-guide distance.
• Even with randomly initialized networks (no training), the method produces adversarial images with similar distance ratios, suggesting the phenomenon is rooted in network architecture rather than learned weights.
• The adversarial representations are not outliers; they lie in high-density regions of the DNN feature space, indicating they are generic and natural-looking in representation space.
• The feature-opt method outperforms feature-linear across all layers, indicating that non-linearities in DNNs are essential for achieving strong representation mimicry.
• Failure cases were observed with handwritten digits and fine-tuned networks on narrow-domain datasets, suggesting sensitivity to input domain, network depth, and receptive field size.