[Paper Review] Random Smoothing Might be Unable to Certify $\ell_\infty$ Robustness for High-Dimensional Images

Avrim Blum, Travis Dick+2 more|arXiv (Cornell University)|Feb 10, 2020

Adversarial Robustness in Machine Learning26 references15 citations

TL;DR

This paper demonstrates that random smoothing cannot achieve non-trivial $ε$-robustness under the $∞$-norm threat model for high-dimensional images using any noise distribution, as the required noise variance per pixel grows with dimension $d$ as $\Omega(d^{1-2/p}\epsilon^2(1-\delta)/\delta^2)$ for $p>2$. This implies that for $p=\infty$, the noise overwhelms the signal, rendering smoothed classifiers trivial and ineffective in practice.

ABSTRACT

We show a hardness result for random smoothing to achieve certified adversarial robustness against attacks in the $\ell_p$ ball of radius $ε$ when $p>2$. Although random smoothing has been well understood for the $\ell_2$ case using the Gaussian distribution, much remains unknown concerning the existence of a noise distribution that works for the case of $p>2$. This has been posed as an open problem by Cohen et al. (2019) and includes many significant paradigms such as the $\ell_\infty$ threat model. In this work, we show that any noise distribution $\mathcal{D}$ over $\mathbb{R}^d$ that provides $\ell_p$ robustness for all base classifiers with $p>2$ must satisfy $\mathbb{E}η_i^2=Ω(d^{1-2/p}ε^2(1-δ)/δ^2)$ for 99% of the features (pixels) of vector $η\sim\mathcal{D}$, where $ε$ is the robust radius and $δ$ is the score gap between the highest-scored class and the runner-up. Therefore, for high-dimensional images with pixel values bounded in $[0,255]$, the required noise will eventually dominate the useful information in the images, leading to trivial smoothed classifiers.

Motivation & Objective

• To investigate whether random smoothing can achieve certified robustness for $\ell_p$-norm attacks when $p>2$, particularly in the $\ell_\infty$ case.
• To identify fundamental limitations of random smoothing in high-dimensional settings where pixel values are bounded in $[0,255]$.
• To derive lower bounds on the noise variance required for $\ell_p$-robustness certification when $p>2$, showing a phase transition at $p=2$.
• To resolve an open problem posed by Cohen et al. (2019) on the existence of noise distributions for $\ell_p$ robustness with $p>2$.

Proposed method

• Derives a lower bound on the second moment $\mathbb{E}[\eta_i^2]$ of noise components $\eta_i$ in the smoothing distribution $\mathcal{D}$ for $\ell_p$-robustness with $p>2$.
• Uses total variation distance between $\mathcal{D}$ and $\mathcal{D}+v$ for $\|v\|_p \leq \epsilon$ to quantify robustness guarantees.
• Applies Markov's inequality and properties of total variation to bound the expected absolute value of noise projections along perturbation directions.
• Constructs $b > d/2$ vectors $v_i$ with $\|v_i\|_p = \epsilon$ and $\|v_i\|_2 = \epsilon \cdot b^{1/2 - 1/p}$ to exploit directional sensitivity in high dimensions.
• Uses orthogonal projection and $\ell_1$-$\ell_2$ norm equivalence to relate the expected $\ell_2$-norm of noise to the lower bound on $\mathbb{E}[|v_i^T \eta|/\|v_i\|_2]$.
• Establishes that $\mathbb{E}[\|\eta\|_2] \geq \frac{\epsilon d^{1-1/p}}{24} \cdot \frac{1-\delta}{\delta}$, leading to $\mathbb{E}[\eta_i^2] = \Omega(d^{1-2/p}\epsilon^2(1-\delta)/\delta^2)$ for 99% of features.

Experimental results

Research questions

• RQ1 Can random smoothing with any noise distribution provide certified $\ell_p$-robustness for $p>2$?
• RQ2 What is the minimum noise variance required per pixel to achieve $\epsilon$-robustness under $\ell_p$-norm perturbations for $p>2$?
• RQ3 Does the required noise scale with dimension $d$ in a way that makes smoothed classifiers impractical for high-dimensional images?
• RQ4 Is there a fundamental barrier to achieving non-trivial $\ell_\infty$-robustness via random smoothing in high-dimensional settings?
• RQ5 Does the phase transition at $p=2$ imply that $\ell_2$-smoothing is uniquely viable among $\ell_p$-norms?

Key findings

• For $p>2$, the required noise variance per pixel grows as $\Omega(d^{1-2/p}\epsilon^2(1-\delta)/\delta^2)$, which increases with dimension $d$.
• When $p=\infty$, the noise variance per pixel scales as $\Omega(d^{1-2/\infty}\epsilon^2(1-\delta)/\delta^2) = \Omega(d\epsilon^2(1-\delta)/\delta^2)$, dominating the signal in high-dimensional images.
• The lower bound on $\mathbb{E}[\|\eta\|_2]$ is $\frac{\epsilon d^{1-1/p}}{24} \cdot \frac{1-\delta}{\delta}$, showing that noise magnitude grows with $d$ for $p>2$.
• The result implies that for $\ell_\infty$ robustness, random smoothing becomes ineffective in practice because the noise overwhelms the input features.
• The phase transition at $p=2$ means that while $\ell_2$-smoothing can use constant noise variance, $\ell_p$-smoothing for $p>2$ requires growing noise, making it unsuitable for high-dimensional data.
• The paper resolves an open problem by showing that no noise distribution can achieve non-trivial $\ell_\infty$ robustness via random smoothing in high-dimensional image settings.