[Paper Review] On the Limitations of Stochastic Pre-processing Defenses

Yue Gao, Ilia Shumailov+2 more|arXiv (Cornell University)|Jun 19, 2022

Adversarial Robustness in Machine Learning5 citations

TL;DR

This paper investigates stochastic pre-processing defenses—randomized input transformations applied before model inference—and demonstrates they are fundamentally limited. It shows most such defenses lack sufficient randomness to resist standard attacks like PGD, and that a trade-off exists between robustness and model invariance to the randomization, undermining their effectiveness. The authors provide theoretical and empirical evidence that randomness alone does not ensure robustness, and future defenses must decouple robustness from invariance to avoid failure.

ABSTRACT

Defending against adversarial examples remains an open problem. A common belief is that randomness at inference increases the cost of finding adversarial inputs. An example of such a defense is to apply a random transformation to inputs prior to feeding them to the model. In this paper, we empirically and theoretically investigate such stochastic pre-processing defenses and demonstrate that they are flawed. First, we show that most stochastic defenses are weaker than previously thought; they lack sufficient randomness to withstand even standard attacks like projected gradient descent. This casts doubt on a long-held assumption that stochastic defenses invalidate attacks designed to evade deterministic defenses and force attackers to integrate the Expectation over Transformation (EOT) concept. Second, we show that stochastic defenses confront a trade-off between adversarial robustness and model invariance; they become less effective as the defended model acquires more invariance to their randomization. Future work will need to decouple these two effects. We also discuss implications and guidance for future research.

Motivation & Objective

• To investigate the fundamental limitations of stochastic pre-processing defenses that rely on random input transformations at inference time.
• To challenge the widely held belief that randomness inherently increases adversarial attack cost.
• To identify and analyze the trade-off between adversarial robustness and model invariance induced by such defenses.
• To provide guidance for future defense design by exposing the pitfalls of current randomization strategies.

Proposed method

• The authors conduct systematic empirical evaluations on multiple stochastic defenses using standard adversarial attacks like PGD without Expectation over Transformation (EOT), to test robustness under minimal adaptive assumptions.
• They analyze the impact of increasing randomization space (e.g., in BaRT) on defense effectiveness, comparing results with and without EOT.
• Theoretical analysis is developed to formalize the trade-off between robustness and invariance, showing that robustness arises from induced variance, which must be reduced to recover utility.
• Empirical validation is performed on standard datasets (e.g., ImageNet, ImageNette) and models, measuring robustness degradation when models are fine-tuned to be invariant to transformations.
• The study evaluates both well-known defenses (e.g., BaRT, DiffPure) and newer ones, including randomized activation pruning and discontinued activation, to assess generalizability.
• Guidelines for future attack and defense design are derived from observed patterns, emphasizing the need to decouple robustness from invariance.

Experimental results

Research questions

• RQ1 Why do many stochastic pre-processing defenses fail against standard attacks like PGD, despite relying on randomness?
• RQ2 What is the role of Expectation over Transformation (EOT) in breaking stochastic defenses, and when is it actually necessary?
• RQ3 How does model invariance to random transformations affect the robustness of stochastic defenses?
• RQ4 Can a theoretical trade-off between robustness and invariance be formally established in stochastic pre-processing defenses?
• RQ5 What design principles should guide future stochastic defenses to avoid the pitfalls of current approaches?

Key findings

• Most stochastic pre-processing defenses are not sufficiently randomized to resist standard attacks like PGD, even without EOT, undermining the assumption that randomness inherently increases attack cost.
• The use of EOT only provides an advantage when defenses have high enough randomness; otherwise, standard attacks already perform well, indicating overestimation of robustness.
• A fundamental trade-off exists between adversarial robustness and model invariance: as models become more invariant to the random transformations, robustness decreases.
• Empirical results show that fine-tuning models on defense-augmented data reduces robustness, confirming the theoretical trade-off.
• Theoretical analysis proves that robustness in such defenses arises from induced variance, which must be reduced to preserve utility, making the defense inherently fragile.
• The findings generalize to other stochastic defenses beyond input transformations, including randomized activation pruning and discontinued activation, suggesting broad implications for defense design.