[Paper Review] Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks

Avi Schwarzschild, Micah Goldblum+3 more|arXiv (Cornell University)|Jun 22, 2020

Adversarial Robustness in Machine LearningComputer Science42 references56 citations

TL;DR

The paper proposes standardized benchmarks for evaluating backdoor and triggerless data poisoning attacks on image classifiers, and shows how experimental design and training setup dramatically affect attack effectiveness.

ABSTRACT

Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference. A recent survey of industry practitioners found that data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. However, it remains unclear exactly how dangerous poisoning methods are and which ones are more effective considering that these methods, even ones with identical objectives, have not been tested in consistent or realistic settings. We observe that data poisoning and backdoor attacks are highly sensitive to variations in the testing setup. Moreover, we find that existing methods may not generalize to realistic settings. While these existing works serve as valuable prototypes for data poisoning, we apply rigorous tests to determine the extent to which we should fear them. In order to promote fair comparison in future work, we develop standardized benchmarks for data poisoning and backdoor attacks.

Motivation & Objective

• Motivate fair, comparable evaluation of data poisoning and backdoor attacks.
• Identify how testing setup, dataset size, and training protocol influence attack success.
• Provide standardized benchmarks and publicly available code for reproducible assessments.

Proposed method

• Define and compare triggerless and backdoor poisoning attacks under a unified framework.
• Standardize data, threat models, and training regimes (transfer learning vs. from-scratch).
• Constrain poisons to remain within an 8/255 l-infinity ball and evaluate across white-box and black-box settings.
• Use randomized target/base image pairs and 100 trials per experiment to quantify variance.
• Assess impact of optimizers (SGD vs. Adam), data augmentation, and victim architectures on attack success.
• Provide pre-trained models and fixed evaluation protocols to enable fair cross-method benchmarking.

Experimental results

Research questions

• RQ1 How do realistic training setups and benchmarks affect the observed efficacy of poisoning attacks?
• RQ2 Do commonly used poison budgets (percentage of data poisoned) reliably predict attack strength across varying dataset sizes and architectures?
• RQ3 How do transfer learning vs. training-from-scratch regimes influence attack success in standardized benchmarks?
• RQ4 What factors (e.g., data augmentation, optimizer, model architecture) most alter the fragility of systems to data poisoning?
• RQ5 Are reported “clean-label” poisons truly clean under standardized evaluation?

Key findings

• Attack effectiveness is highly sensitive to training setup and dataset context.
• SGD with data augmentation markedly reduces attack success compared to Adam and no augmentation (e.g., FC and CP drop to 51.00% and 19.09% in the augmented SGD setting).
• Victim architecture matters; some attacks are far less effective on ResNet-18 than on AlexNet variants.
• Many so-called clean-label poisons exhibit perceptible artifacts under common perturbation radii, challenging the “clean” label claim.
• Budget percentage alone is insufficient; dataset size fundamentally changes attack effectiveness and attack curves can cross between methods.
• Black-box transfer attacks show substantially lower success rates than white-box baselines (often <20%).