[Paper Review] Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training?

Ali Shafahi, Amin Ghiasi+2 more|arXiv (Cornell University)|Oct 25, 2019

Adversarial Robustness in Machine Learning19 references34 citations

TL;DR

The paper shows that simple regularizers (label smoothing and logit squeezing) combined with Gaussian noise can achieve robustness comparable to or greater than adversarial training, without using adversarial examples.

ABSTRACT

Adversarial training is one of the strongest defenses against adversarial attacks, but it requires adversarial examples to be generated for every mini-batch during optimization. The expense of producing these examples during training often precludes adversarial training from use on complex image datasets. In this study, we explore the mechanisms by which adversarial training improves classifier robustness, and show that these mechanisms can be effectively mimicked using simple regularization methods, including label smoothing and logit squeezing. Remarkably, using these simple regularization methods in combination with Gaussian noise injection, we are able to achieve strong adversarial robustness -- often exceeding that of adversarial training -- using no adversarial examples.

Motivation & Objective

• Understand how adversarial training yields robustness and what it accomplishes in terms of logits and gradients.
• Explore whether regularization methods like label smoothing and logit squeezing can mimic adversarial training effects.
• Assess the role of Gaussian noise augmentation in enhancing robustness when used with simple regularizers.
• Demonstrate empirical robustness and compare with adversarial training on standard datasets (MNIST, CIFAR-10/100).

Proposed method

• Decompose adversarial robustness using a linearized model relating logit gap, gradient gap, and perturbation size.
• Apply label smoothing to reduce logit gaps and mimic reduced confidence.
• Apply logit squeezing to penalize large logits and shrink gradients.
• Augment training data with Gaussian noise to regularize off-manifold inputs.
• Combine regularizers with Gaussian augmentation and evaluate against FGSM and iterative PGD attacks.
• Compare performance to standard adversarial training (e.g., Madry et al.) on MNIST, CIFAR-10, and CIFAR-100.

Experimental results

Research questions

• RQ1 Can label smoothing and logit squeezing replicate key robustness effects of adversarial training?
• RQ2 What is the role of Gaussian noise augmentation in enabling these regularizers to match or surpass adversarial training?
• RQ3 How do these methods perform under white-box versus black-box adversarial threats on standard benchmarks?

Key findings

• Label smoothing plus Gaussian augmentation yields high robustness, with CIFAR-10 achieving over 73% accuracy against black-box iterative attacks, compared to 64% for a state-of-the-art adversarially trained model.
• In the white-box setting, classifiers trained with logit squeezing and label smoothing reach approximately 50% accuracy on iterative attacks, versus about 47% for adversarial training.
• Aggressive logit squeezing (e.g., beta=10) with Gaussian noise can outperform Madry et al.’s adversarial training on CIFAR-10 under white-box PGD-20 attacks, while also delivering better clean accuracy in some cases.
• Label smoothing by itself can slightly reduce robustness, but when paired with Gaussian augmentation it markedly reduces the gradient gap and aligns gradients, boosting robustness.
• Logit squeezing with Gaussian noise increases the logit gap in some configurations and reduces gradient magnitude, producing robustness patterns similar to adversarial training; the combination can be especially effective on CIFAR-10/100.