[Paper Review] Towards Deep Neural Network Architectures Robust to Adversarial Examples

Shixiang Gu, Luca Rigazio|arXiv (Cornell University)|Dec 11, 2014

Adversarial Robustness in Machine Learning19 references632 citations

TL;DR

This paper proposes Deep Contractive Networks (DCNs), a novel training procedure that incorporates a smoothness penalty inspired by contractive autoencoders to enhance deep neural network robustness against adversarial examples. By enforcing layer-wise invariance to small input perturbations, DCNs increase the minimum distortion required to generate adversarial examples—demonstrating a 25–30% increase in adversarial distortion compared to standard networks—while maintaining high test accuracy on clean data.

ABSTRACT

Recent work has shown deep neural networks (DNNs) to be highly susceptible to well-designed, small perturbations at the input layer, or so-called adversarial examples. Taking images as an example, such distortions are often imperceptible, but can result in 100% mis-classification for a state of the art DNN. We study the structure of adversarial examples and explore network topology, pre-processing and training strategies to improve the robustness of DNNs. We perform various experiments to assess the removability of adversarial examples by corrupting with additional noise and pre-processing with denoising autoencoders (DAEs). We find that DAEs can remove substantial amounts of the adversarial noise. How- ever, when stacking the DAE with the original DNN, the resulting network can again be attacked by new adversarial examples with even smaller distortion. As a solution, we propose Deep Contractive Network, a model with a new end-to-end training procedure that includes a smoothness penalty inspired by the contractive autoencoder (CAE). This increases the network robustness to adversarial examples, without a significant performance penalty.

Motivation & Objective

• To address the vulnerability of deep neural networks (DNNs) to small, imperceptible input perturbations that cause misclassification, known as adversarial examples.
• To investigate whether adversarial examples can be removed via denoising techniques like denoising autoencoders (DAEs), and whether such removal improves overall robustness.
• To explore whether the root cause of adversarial vulnerability lies in the training procedure rather than network architecture, and to design a training objective that enforces local invariance.
• To develop a new end-to-end training framework that explicitly learns robust, flat representations around training data by incorporating contractive penalties at each layer.
• To evaluate whether combining contractive penalties with data augmentation (e.g., Gaussian noise) further enhances adversarial robustness without sacrificing clean accuracy.

Proposed method

• Proposes Deep Contractive Networks (DCNs), a new training procedure that applies a smoothness penalty inspired by contractive autoencoders (CAE) to each hidden layer of a DNN.
• The penalty is computed as the Frobenius norm of the Jacobian of the hidden representation with respect to the input, encouraging the network to learn flat, invariant representations near training data points.
• The method integrates the contractive penalty into the standard backpropagation process, enabling end-to-end training while preserving high classification accuracy on clean data.
• Employs a layer-wise contractive objective that regularizes each hidden layer to be insensitive to small input perturbations, effectively making the network more robust to adversarial noise.
• Validates the approach by training models with and without the contractive penalty and comparing adversarial robustness using the L2 distortion of adversarial examples.
• Augments the DCN framework with Gaussian input noise during training to further increase the minimum distortion required for successful adversarial attacks.

Experimental results

Research questions

• RQ1 Can adversarial examples be effectively removed using denoising autoencoders (DAEs), and does this lead to a more robust overall network?
• RQ2 Does the sensitivity of DNNs to adversarial examples stem from architectural design or from flaws in the training objective function?
• RQ3 Can a contractive penalty applied at each layer improve the network's robustness to adversarial perturbations without degrading clean accuracy?
• RQ4 How does the minimum L2 distortion of adversarial examples change when using a Deep Contractive Network compared to a standard DNN?
• RQ5 Can combining the contractive penalty with data augmentation (e.g., Gaussian noise) further increase the robustness of the model to adversarial attacks?

Key findings

• Denosing autoencoders (DAEs) can successfully remove a substantial portion of adversarial noise from inputs, but the resulting stacked DAE-DNN model remains vulnerable to new, even smaller adversarial examples.
• The Deep Contractive Network (DCN) model increases the average L2 distortion of adversarial examples by 25–30% compared to standard networks: for example, from 0.084 to 0.107 in the N100-100-10 model.
• DCNs achieve high clean test accuracy (e.g., 0.9% error on CIFAR-10) while simultaneously increasing adversarial distortion, indicating that robustness can be enhanced without performance trade-offs.
• The DCN with additional Gaussian noise augmentation (DCN+GN,L1,σ=0.1) achieves an adversarial distortion of 0.108, outperforming both standard networks and noise-augmented models without the contractive penalty.
• The contractive penalty effectively propagates invariance through the network, as evidenced by the consistent increase in adversarial distortion across multiple architectures (e.g., N200-200-10, ConvNet).
• The results suggest that adversarial vulnerability is primarily due to the training objective rather than network topology, and that explicit invariance learning via contractive penalties is a viable path to robustness.