[Paper Review] Stronger Data Poisoning Attacks Break Data Sanitization Defenses

Pang Wei Koh, Jacob Steinhardt+1 more|arXiv (Cornell University)|Nov 2, 2018

Adversarial Robustness in Machine LearningComputer Science99 citations

TL;DR

The paper develops three coordinated data poisoning attacks that evade common data sanitization defenses and significantly degrade test accuracy with only 3% poisoned data. It shows these attacks remain effective against anomaly detectors based on nearest neighbors, training loss, SVD, and centroid distance.

ABSTRACT

Machine learning models trained on data from the outside world can be corrupted by data poisoning attacks that inject malicious points into the models' training sets. A common defense against these attacks is data sanitization: first filter out anomalous training points before training the model. In this paper, we develop three attacks that can bypass a broad range of common data sanitization defenses, including anomaly detectors based on nearest neighbors, training loss, and singular-value decomposition. By adding just 3% poisoned data, our attacks successfully increase test error on the Enron spam detection dataset from 3% to 24% and on the IMDB sentiment classification dataset from 12% to 29%. In contrast, existing attacks which do not explicitly account for these data sanitization defenses are defeated by them. Our attacks are based on two ideas: (i) we coordinate our attacks to place poisoned points near one another, and (ii) we formulate each attack as a constrained optimization problem, with constraints designed to ensure that the poisoned points evade detection. As this optimization involves solving an expensive bilevel problem, our three attacks correspond to different ways of approximating this problem, based on influence functions; minimax duality; and the Karush-Kuhn-Tucker (KKT) conditions. Our results underscore the need to develop more robust defenses against data poisoning attacks.

Motivation & Objective

• Motivate and formalize data poisoning risks when defenders use data sanitization to filter anomalous training points.
• Show that coordinated poisoning can evade diverse anomaly detectors and degrade model performance.
• Propose three attack frameworks that leverage concentration, constrained optimization, and decoy parameters to bypass defenses.
• Demonstrate substantial test-error increases on real datasets under realistic defense assumptions.

Proposed method

• Formulate attacks as constrained optimization problems where poisoned points must evade the defender's anomaly detector.
• Concentrate poisoned points in a few locations to defeat sensitive anomaly detectors while maintaining effectiveness.
• Develop three attack variants—Influence, KKT, and Minimax—to approximate the bilevel optimization needed for poisoning.
• Use decoy parameters to decouple attacker optimization from the defender's learned model, enabling efficient attack computation.
• Provide a randomized rounding approach to handle integer-valued input domains and ensure attack concentration.
• Offer iterative optimization to refine both the poisoned set and the anomaly detector parameters.

Experimental results

Research questions

• RQ1 Can data sanitization defenses reliably detect and discard poisoned data when attackers coordinate multiple points?
• RQ2 What attack strategies enable evading diverse anomaly detectors such as k-NN, L2, slab, loss-based, and SVD defenses?
• RQ3 How effective are coordinated poisoning attacks in increasing test error on standard datasets under defense constraints?
• RQ4 Do attack techniques like concentration and decoy-parameter optimization generalize across two-class and multi-class classifiers under convex losses?
• RQ5 What computational strategies make bilevel poisoning optimization tractable on realistic datasets?

Key findings

• Attacks can increase test error from 3% to 24% on Enron spam with 3% poisoned data even under sanitization.
• Attacks can increase test error from 12% to 29% on IMDB sentiment with 3% poisoned data even under sanitization.
• Concentrated poisoning points evade highly sensitive anomaly detectors by clustering in a few locations.
• Two poisoned points suffice for some 2-class SVMs or logistic regression to realize effective attacks under convex losses.
• Three attack formulations (Influence, KKT, Minimax) balance computational efficiency and defense evasion.
• Regularization can paradoxically increase defender vulnerability by reducing fit to the small poisoned subset.