[Paper Review] Denoising Dictionary Learning Against Adversarial Perturbations

John Mitro, Derek Bridge+1 more|arXiv (Cornell University)|Jan 7, 2018

Adversarial Robustness in Machine Learning3 citations

TL;DR

This paper proposes denoising dictionary learning (DDL) as a defense against adversarial perturbations in deep neural networks. By learning a redundant, overcomplete dictionary to reconstruct clean images from adversarially perturbed inputs, DDL achieves high-confidence correct predictions on MNIST and CIFAR10, significantly improving robustness across diverse architectures like MLP, CNN, ResNet, and HRNN.

ABSTRACT

We propose denoising dictionary learning (DDL), a simple yet effective technique as a protection measure against adversarial perturbations. We examined denoising dictionary learning on MNIST and CIFAR10 perturbed under two different perturbation techniques, fast gradient sign (FGSM) and jacobian saliency maps (JSMA). We evaluated it against five different deep neural networks (DNN) representing the building blocks of most recent architectures indicating a successive progression of model complexity of each other. We show that each model tends to capture different representations based on their architecture. For each model we recorded its accuracy both on the perturbed test data previously misclassified with high confidence and on the denoised one after the reconstruction using dictionary learning. The reconstruction quality of each data point is assessed by means of PSNR (Peak Signal to Noise Ratio) and Structure Similarity Index (SSI). We show that after applying (DDL) the reconstruction of the original data point from a noisy

Motivation & Objective

• To address the vulnerability of deep neural networks to adversarial perturbations that cause high-confidence misclassifications.
• To develop a defense mechanism that is robust, flexible, and invertible, avoiding the need to store original and transformed data.
• To evaluate the effectiveness of dictionary learning across diverse DNN architectures with varying complexity and topology.
• To assess reconstruction quality using PSNR and SSIM, ensuring semantic and structural fidelity post-denoising.
• To demonstrate that DDL can restore clean inputs from heavily perturbed test samples without requiring access to clean training data during inference.

Proposed method

• Denoising dictionary learning (DDL) is applied to reconstruct clean images from adversarially perturbed inputs using a learned overcomplete dictionary.
• The method solves a sparse coding problem to represent each perturbed image as a sparse linear combination of dictionary atoms.
• The reconstruction is obtained by projecting the perturbed input onto the learned dictionary and reconstructing using the sparse coefficients.
• The dictionary is trained on clean data only, enabling it to generalize to perturbed test samples without requiring noisy training examples.
• The approach is architecture-agnostic and can be integrated into any supervised learning pipeline without modifying the underlying DNN.
• Reconstruction quality is evaluated using PSNR and SSIM to ensure structural and signal fidelity post-denoising.

Experimental results

Research questions

• RQ1 Can denoising dictionary learning effectively restore clean images from adversarially perturbed inputs under FGSM and JSMA attacks?
• RQ2 How does DDL perform across diverse deep neural network architectures with varying complexity and topology?
• RQ3 To what extent does DDL improve classification accuracy on perturbed test data compared to baseline models?
• RQ4 How does the reconstruction quality, measured by PSNR and SSIM, correlate with improved model robustness?
• RQ5 Can DDL serve as a flexible, invertible, and computationally efficient defense mechanism without requiring storage of original and transformed data?

Key findings

• After applying DDL, classification accuracy on FGSM-perturbed MNIST improved from 12.80% to 82%, and on CIFAR10 from 15.06% to 68.29%.
• For JSMA-perturbed data, accuracy on MNIST increased from 53.02% to 60% and on CIFAR10 from 14.59% to 67.23% across models.
• The auto-encoder and ResNet models showed the most significant gains, with accuracy improvements of 29.45% and 23.01% respectively on FGSM-MNIST.
• Reconstruction quality was high, with PSNR and SSIM values indicating strong structural and signal fidelity in denoised images.
• Models with high sensitivity to small image details, such as ResNet, were more vulnerable to perturbations but benefited significantly from DDL.
• The method successfully restored clean inputs even under strong perturbations, enabling high-confidence correct predictions post-denoising.