[Paper Review] Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression

Masanori Yamada, Sekitoshi Kanai+5 more|arXiv (Cornell University)|Feb 5, 2021

Adversarial Robustness in Machine Learning29 references4 citations

TL;DR

This paper theoretically and experimentally demonstrates that adversarial training with $L_2$-constrained perturbations sharpens the weight loss landscape in logistic regression, primarily due to noise aligned to increase loss. The sharpness increases with adversarial noise magnitude, and this effect is more pronounced than with random noise, explaining degraded generalization in robust models.

ABSTRACT

Adversarial training is actively studied for learning robust models against adversarial examples. A recent study finds that adversarially trained models degenerate generalization performance on adversarial examples when their weight loss landscape, which is loss changes with respect to weights, is sharp. Unfortunately, it has been experimentally shown that adversarial training sharpens the weight loss landscape, but this phenomenon has not been theoretically clarified. Therefore, we theoretically analyze this phenomenon in this paper. As a first step, this paper proves that adversarial training with the L2 norm constraints sharpens the weight loss landscape in the linear logistic regression model. Our analysis reveals that the sharpness of the weight loss landscape is caused by the noise aligned in the direction of increasing the loss, which is used in adversarial training. We theoretically and experimentally confirm that the weight loss landscape becomes sharper as the magnitude of the noise of adversarial training increases in the linear logistic regression model. Moreover, we experimentally confirm the same phenomena in ResNet18 with softmax as a more general case.

Motivation & Objective

• To theoretically clarify why adversarial training sharpens the weight loss landscape in logistic regression.
• To analyze the role of adversarial noise direction and magnitude in shaping loss landscape sharpness.
• To compare adversarial noise effects with random noise in loss landscape curvature.
• To extend findings to nonlinear models like ResNet18 with softmax.
• To explain the generalization gap in adversarially trained models via loss landscape sharpness.

Proposed method

• Theoretical analysis uses Hessian eigenvalues to define loss landscape sharpness in linear logistic regression.
• Adversarial examples are decomposed into magnitude and direction components for analysis.
• The Hessian of the loss function is derived under $L_2$-norm constrained adversarial training.
• Optimal weight condition is applied to isolate the effect of adversarial noise on curvature.
• Theoretical results are validated via experiments on MNIST2 and ResNet18 with varying noise magnitudes.
• Comparative experiments assess loss landscape sharpness under adversarial vs. random noise.

Experimental results

Research questions

• RQ1 Does adversarial training with $L_2$-norm constraints systematically increase the sharpness of the weight loss landscape in logistic regression?
• RQ2 How does the magnitude of adversarial noise affect the curvature of the loss landscape?
• RQ3 Is the sharpness increase due to adversarial noise distinct from that caused by random noise?
• RQ4 Does the same phenomenon hold in nonlinear models like ResNet18 with softmax?
• RQ5 Can the generalization gap in adversarially trained models be attributed to increased loss landscape sharpness?

Key findings

• Theoretical analysis proves that the Hessian eigenvalues of the loss landscape scale linearly with the norm of adversarial noise in $L_2$-constrained logistic regression.
• The sharpness increase is caused by adversarial noise being aligned to maximize loss, not random perturbations.
• Experiments on MNIST2 confirm that larger adversarial noise leads to significantly sharper loss landscapes.
• In ResNet18 with softmax, increasing adversarial noise magnitude results in progressively sharper loss landscapes.
• Adversarial noise sharpens the loss landscape much more than random noise, which shows minimal curvature increase.
• The study concludes that loss landscape sharpness is a key factor in the generalization gap of adversarially trained models.