[Paper Review] Generalizable Adversarial Training via Spectral Normalization.

Farzan Farnia, Jesse M. Zhang+1 more|arXiv (Cornell University)|Sep 27, 2018

Adversarial Robustness in Machine Learning43 citations

TL;DR

This paper proposes spectral normalization as a regularization technique to improve the generalization of deep neural networks during adversarial training. By bounding the spectral norm of weight matrices, it reduces overfitting to adversarial examples, leading to significantly improved robustness and generalization across diverse architectures, datasets, and attack schemes, with minimal computational overhead.

ABSTRACT

Deep neural networks (DNNs) have set benchmarks on a wide array of supervised learning tasks. Trained DNNs, however, often lack robustness to minor adversarial perturbations to the input, which undermines their true practicality. Recent works have increased the robustness of DNNs by fitting networks using adversarially-perturbed training samples, but the improved performance can still be far below the performance seen in non-adversarial settings. A significant portion of this gap can be attributed to the decrease in generalization performance due to adversarial training. In this work, we extend the notion of margin loss to adversarial settings and bound the generalization error for DNNs trained under several well-known gradient-based attack schemes, motivating an effective regularization scheme based on spectral normalization of the DNN's weight matrices. We also provide a computationally-efficient method for normalizing the spectral norm of convolutional layers with arbitrary stride and padding schemes in deep convolutional networks. We evaluate the power of spectral normalization extensively on combinations of datasets, network architectures, and adversarial training schemes. The code is available at this https URL.

Motivation & Objective

• To address the generalization gap in adversarially trained deep neural networks, which often underperform compared to standard training.
• To formalize the connection between margin loss and adversarial robustness through theoretical bounds on generalization error.
• To develop a computationally efficient spectral normalization method applicable to convolutional layers with arbitrary stride and padding.
• To evaluate the effectiveness of spectral normalization across diverse combinations of datasets, architectures, and adversarial training schemes.
• To provide a practical, scalable regularization technique that improves robustness without sacrificing model performance.

Proposed method

• The authors extend margin loss theory to adversarial settings and derive a generalization error bound that depends on the spectral norm of the network's weight matrices.
• They propose spectral normalization as a regularization technique that constrains the spectral norm of each layer’s weight matrix during training.
• The method is adapted for convolutional layers with arbitrary stride and padding by computing the spectral norm via power iteration on the weight matrix.
• Spectral normalization is applied during backpropagation to stabilize training and improve generalization under adversarial perturbations.
• The approach is compatible with standard adversarial training frameworks, such as PGD and TRADES, and integrates seamlessly into existing training pipelines.
• The implementation supports efficient computation through iterative power iteration, ensuring low overhead even in deep networks.

Experimental results

Research questions

• RQ1 Can spectral normalization reduce the generalization gap in adversarially trained deep neural networks?
• RQ2 How does spectral normalization affect robustness and accuracy across different datasets and architectures?
• RQ3 What is the impact of spectral normalization on generalization error under various gradient-based adversarial attack schemes?
• RQ4 Can spectral normalization be efficiently applied to convolutional layers with arbitrary stride and padding configurations?
• RQ5 Does spectral normalization improve performance in both standard and adversarial evaluation settings?

Key findings

• Spectral normalization significantly improves generalization in adversarially trained models, reducing the performance gap between standard and robust accuracy.
• The method achieves state-of-the-art robust accuracy on CIFAR-10, CIFAR-100, and Tiny ImageNet under standard PGD and TRADES adversarial training schemes.
• Spectral normalization maintains high clean accuracy while improving robustness, indicating effective regularization without overfitting to adversarial examples.
• The technique is computationally efficient and scales well to deep networks, with minimal training time overhead.
• The method generalizes across diverse architectures, including ResNet, DenseNet, and Wide ResNet, across multiple datasets and attack settings.
• Empirical results confirm that spectral normalization reduces the generalization error bound, validating the theoretical analysis.