[Paper Review] Provable tradeoffs in adversarially robust classification

Edgar Dobriban, Hamed Hassani+2 more|arXiv (Cornell University)|Jun 9, 2020

Advanced Statistical Methods and Models4 citations

TL;DR

This paper derives provably optimal robust classifiers for two- and three-class Gaussian mixture models under $ε$-bounded $€_2$ and $€_\infty$ adversarial perturbations, revealing a fundamental tradeoff between standard and robust accuracy that worsens with class imbalance. Using tools from robust isoperimetry and Gaussian measure concentration, it establishes exact Bayes-optimal solutions and finite-sample convergence rates for empirical robust risk minimization.

ABSTRACT

It is well known that machine learning methods can be vulnerable to adversarially-chosen perturbations of their inputs. Despite significant progress in the area, foundational open problems remain. In this paper, we address several key questions. We derive exact and approximate Bayes-optimal robust classifiers for the important setting of two- and three-class Gaussian classification problems with arbitrary imbalance, for $\ell_2$ and $\ell_\infty$ adversaries. In contrast to classical Bayes-optimal classifiers, determining the optimal decisions here cannot be made pointwise and new theoretical approaches are needed. We develop and leverage new tools, including recent breakthroughs from probability theory on robust isoperimetry, which, to our knowledge, have not yet been used in the area. Our results reveal fundamental tradeoffs between standard and robust accuracy that grow when data is imbalanced. We also show further results, including an analysis of classification calibration for convex losses in certain models, and finite sample rates for the robust risk.

Motivation & Objective

• To understand the fundamental tradeoffs between standard and robust accuracy in adversarially robust classification under class imbalance.
• To derive exact Bayes-optimal robust classifiers for two- and three-class Gaussian models with arbitrary class imbalance.
• To analyze the optimization landscape of robust risk and its connection to convex surrogate losses.
• To provide finite-sample convergence rates for empirical robust risk minimization in low-dimensional settings.

Proposed method

• Leveraged robust isoperimetry and Gaussian measure concentration to characterize optimal decision boundaries under $€_2$ and $€_\infty$ perturbations.
• Derived exact Bayes-optimal classifiers by solving for extremal sets under Gaussian isoperimetry, showing that optimal decisions depend on global data structure, not pointwise likelihood.
• Used the Dvoretzky–Kiefer–Wolfowitz inequality to establish uniform concentration of empirical robust risk over classifiers with at most $k$ interval regions in one dimension.
• Proved that the optimizers of convex surrogate losses (e.g., hinge, logistic) coincide with the nonconvex robust 0-1 loss under the considered models.
• Analyzed the robust risk as a function of perturbation radius $\varepsilon$, class means, and class priors to quantify tradeoffs.
• Established finite-sample convergence rates of $O(k/\sqrt{n})$ for empirical robust risk over classifiers with at most $2k$ interval regions in one-dimensional data.

Experimental results

Research questions

• RQ1 What is the exact form of the Bayes-optimal robust classifier for two- and three-class Gaussian models under $€_2$ and $€_\infty$ adversarial attacks with class imbalance?
• RQ2 How does class imbalance affect the fundamental tradeoff between standard accuracy and robust accuracy in adversarial settings?
• RQ3 Do convex surrogate losses in robust training yield the same optimal classifier as the nonconvex robust 0-1 loss in Gaussian mixture models?
• RQ4 What are the finite-sample convergence rates of empirical robust risk minimization for linear and piecewise-constant classifiers in one-dimensional settings?
• RQ5 How does the geometry of $€_p$-ball expansions of decision regions affect the robust risk in Gaussian models?

Key findings

• The Bayes-optimal robust classifier for Gaussian mixtures under $€_2$ and $€_\infty$ perturbations cannot be determined pointwise and requires global geometric analysis via robust isoperimetry.
• A fundamental tradeoff between standard and robust accuracy exists and intensifies with increasing class imbalance, making simultaneous optimization impossible.
• For one-dimensional data, the empirical robust risk converges uniformly at rate $O(k/\sqrt{n})$ over classifiers with at most $2k$ interval regions.
• The optimizers of convex surrogate losses (e.g., hinge, logistic) coincide with the nonconvex robust 0-1 loss under the considered Gaussian models, implying equivalence in optimal decision rules.
• The $€_\varepsilon$-expansion of half-spaces remains a half-space, enabling VC-dimension-based uniform concentration arguments for linear classifiers.
• The robust risk for classifiers with $k$-interval decision regions concentrates uniformly at rate $O(k/\sqrt{n})$, with high probability $1 - 4\exp(-2n\delta^2/k^2)$.