[Paper Review] Exploring the Hyperparameter Landscape of Adversarial Robustness

Evelyn Duesterwald, Anupama Murthi+3 more|arXiv (Cornell University)|May 9, 2019

Adversarial Robustness in Machine Learning21 references4 citations

TL;DR

This paper investigates the complex relationship between adversarial training hyperparameters and model robustness, demonstrating that robustness surfaces are highly sensitive and non-linear. It proposes a hyperparameter optimization (HPO) framework using accuracy budgets to automatically navigate the robustness-accuracy trade-off, achieving near-optimal robustness with minimal accuracy loss on MNIST and SVHN.

ABSTRACT

Adversarial training shows promise as an approach for training models that are robust towards adversarial perturbation. In this paper, we explore some of the practical challenges of adversarial training. We present a sensitivity analysis that illustrates that the effectiveness of adversarial training hinges on the settings of a few salient hyperparameters. We show that the robustness surface that emerges across these salient parameters can be surprisingly complex and that therefore no effective one-size-fits-all parameter settings exist. We then demonstrate that we can use the same salient hyperparameters as tuning knob to navigate the tension that can arise between robustness and accuracy. Based on these findings, we present a practical approach that leverages hyperparameter optimization techniques for tuning adversarial training to maximize robustness while keeping the loss in accuracy within a defined budget.

Motivation & Objective

• To understand the sensitivity of adversarial robustness to key hyperparameters in adversarial training.
• To address the challenge of finding optimal hyperparameter configurations due to complex, non-linear robustness surfaces.
• To mitigate the trade-off between robustness and clean accuracy by introducing an accuracy budget in hyperparameter tuning.
• To develop an automated, principled approach for practitioners to achieve robust models without manual hyperparameter tuning.
• To evaluate the effectiveness of HPO strategies (grid, random, TPE) in navigating the robustness-accuracy trade-off under accuracy constraints.

Proposed method

• Conduct sensitivity analysis on salient hyperparameters: PGD attack ratio and perturbation magnitude (ε) in adversarial training.
• Model adversarial training as a multi-objective optimization problem with robustness as the primary objective and accuracy as a constrained budget.
• Apply three HPO strategies—grid search, random search, and Tree-structured Parzen Estimator (TPE)—to explore the hyperparameter space.
• Incorporate an accuracy budget β to restrict acceptable accuracy loss, guiding HPO toward configurations that balance robustness and clean accuracy.
• Use PGD-based adversarial attacks to evaluate robustness (AdvAcc) and standard test accuracy (TestAcc) across different hyperparameter settings.
• Repeat experiments 40 times per configuration to assess consistency and success rate of HPO methods under tight accuracy constraints.

Experimental results

Research questions

• RQ1 How sensitive is adversarial robustness to variations in key hyperparameters like PGD ratio and ε?
• RQ2 What is the structure of the robustness surface across hyperparameter space, and how complex is it?
• RQ3 Can hyperparameter optimization techniques effectively navigate the robustness-accuracy trade-off in adversarial training?
• RQ4 How does imposing an accuracy budget affect the ability of HPO methods to find optimal hyperparameter configurations?
• RQ5 Can automated HPO strategies outperform manual tuning in achieving high robustness with minimal accuracy degradation?

Key findings

• The robustness surface across hyperparameters (ratio and ε) is highly complex and non-linear, precluding one-size-fits-all hyperparameter settings.
• Grid search achieved 93.4% adversarial accuracy on MNIST and 40.0% on SVHN with no accuracy budget, but at the cost of significant clean accuracy drops (e.g., 70.9% on SVHN).
• With an accuracy budget of β=1.6% on MNIST, grid search found a configuration achieving 92.7% adversarial accuracy while preserving 98.01% clean accuracy.
• On SVHN, an accuracy budget of β=14.0% enabled a configuration with 38.3% adversarial accuracy and 85.6% clean accuracy—significantly better than the 70.9% clean accuracy in the unbounded case.
• Random search and TPE showed consistent performance across 40 repetitions, with TPE slightly outperforming random search due to surrogate modeling, though both were effective under accuracy constraints.
• For very tight accuracy budgets, success rates of random search and TPE decreased, indicating the need for multiple runs to reliably find feasible solutions.