[Paper Review] Double Backpropagation for Training Autoencoders against Adversarial Attack

Chengjin Sun, Sizhe Chen+1 more|arXiv (Cornell University)|Mar 4, 2020

Adversarial Robustness in Machine Learning38 references4 citations

TL;DR

This paper proposes double backpropagation (DBP) to enhance the robustness of autoencoders against adversarial attacks by constraining the gradient of the reconstruction output with respect to input perturbations. By smoothing the encoder's gradient and combining it with Gaussian Mixture Models (GMMs) in the latent space, the method achieves robust autoencoders that resist both white-box and black-box adversarial attacks while maintaining high reconstruction fidelity and enabling stable image transitions.

ABSTRACT

Deep learning, as widely known, is vulnerable to adversarial samples. This paper focuses on the adversarial attack on autoencoders. Safety of the autoencoders (AEs) is important because they are widely used as a compression scheme for data storage and transmission, however, the current autoencoders are easily attacked, i.e., one can slightly modify an input but has totally different codes. The vulnerability is rooted the sensitivity of the autoencoders and to enhance the robustness, we propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW. We restrict the gradient from the reconstruction image to the original one so that the autoencoder is not sensitive to trivial perturbation produced by the adversarial attack. After smoothing the gradient by DBP, we further smooth the label by Gaussian Mixture Model (GMM), aiming for accurate and robust classification. We demonstrate in MNIST, CelebA, SVHN that our method leads to a robust autoencoder resistant to attack and a robust classifier able for image transition and immune to adversarial attack if combined with GMM.

Motivation & Objective

• To address the vulnerability of autoencoders to adversarial attacks that produce drastically different reconstructions from small input perturbations.
• To investigate whether double backpropagation (DBP) can reduce the sensitivity of autoencoders to input perturbations by controlling the gradient of the reconstruction output.
• To explore the use of Gaussian Mixture Models (GMMs) in the latent space to stabilize and smooth the classifier output for improved robustness.
• To evaluate the transferability and robustness of image transition under adversarial conditions when using DBP-regularized autoencoders.
• To demonstrate that combining DBP with GMM leads to a classifier that is resilient to both white-box and black-box adversarial attacks.

Proposed method

• Apply double backpropagation (DBP) to constrain the gradient of the reconstruction output with respect to input perturbations, specifically limiting ∂x′/∂x to reduce sensitivity.
• Introduce a loss function that penalizes large gradients in the encoder by minimizing the magnitude of ∂x′/∂x during training, using a dual backpropagation mechanism.
• Train variational autoencoders (VAE) and DRAW autoencoders with DBP to ensure smooth latent representations and stable reconstructions under small input perturbations.
• Model the distribution of latent codes using Gaussian Mixture Models (GMMs) to provide smooth, continuous decision boundaries in the latent space.
• Use the GMM-regularized latent space to enable image transition by following the gradient of the classifier in the latent space, ensuring coherent and meaningful transformations.
• Evaluate robustness against FGSM and PGD attacks by comparing accuracy of DBP+GMM classifiers with standard classifiers on MNIST and SVHN datasets.

Experimental results

Research questions

• RQ1 Can double backpropagation effectively reduce the sensitivity of autoencoders to small input perturbations, thereby improving robustness against adversarial attacks?
• RQ2 Does combining DBP with GMM in the latent space enhance the robustness of the resulting classifier to both white-box and black-box adversarial attacks?
• RQ3 To what extent does the use of DBP preserve reconstruction quality while improving adversarial robustness in autoencoders?
• RQ4 Can image transition be achieved stably and coherently in the input space when using a DBP-regularized autoencoder with GMM-based classification?
• RQ5 How does the performance of a DBP+GMM classifier compare to standard deep networks under adversarial attacks on MNIST and SVHN?

Key findings

• The DBP-regularized autoencoder achieves robustness against adversarial attacks, with the reconstruction of adversarial inputs remaining close to the original, even under PGD and FGSM attacks.
• On MNIST, the DBP+GMM classifier maintains 93.56% accuracy under PGD attack, compared to only 8.12% for LeNet and 56.64% for MLP, demonstrating strong robustness.
• On SVHN, the DBP+GMM classifier achieves 72.6% accuracy under PGD attack, significantly outperforming the 1.8% accuracy of the baseline CvNet model.
• The classifier with DBP+GMM is robust against black-box attacks, achieving 92.93% accuracy on MNIST and 88.77% on SVHN when adversarial examples are generated from other networks.
• Image transition using the DBP+GMM model produces coherent and meaningful transformations in the input space, with visible features of the target image emerging as confidence increases.
• The method preserves high reconstruction fidelity, as evidenced by low reconstruction error and stable performance across MNIST, CelebA, and SVHN datasets.