[Paper Review] Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

Ali Shafahi, Wei Huang+5 more|arXiv (Cornell University)|Apr 3, 2018

Adversarial Robustness in Machine LearningComputer Science21 references249 citations

TL;DR

This paper proposes targeted clean-label poisoning attacks on neural networks that manipulate model behavior on specific test instances without degrading overall performance. By optimizing poison images to align with target features in the network's hidden layers—using watermarking and diverse base images—it achieves 100% success in transfer learning and up to 60% success in end-to-end training with ~50 poisons.

ABSTRACT

Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a $ extit{specific}$ test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple ($\approx$50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.

Motivation & Objective

• To develop targeted poisoning attacks that manipulate classifier behavior on specific test instances without degrading overall accuracy.
• To design attacks that use clean-label training data—labeled correctly by human or automated systems—making them undetectable by standard data quality checks.
• To enable real-world feasibility by allowing attackers to inject poisons via public data sources (e.g., web-scraped images) without control over labeling.
• To overcome the challenge of end-to-end training, where deep networks' expressive features make poisoning harder than in transfer learning.
• To demonstrate that even a small number of carefully crafted, watermarked poisons can reliably mislead models into misclassifying specific target instances.

Proposed method

• Uses optimization (Algorithm 1) to generate poison images that align with the target image in the feature space of the neural network.
• Applies a watermarking technique by blending the target image with base images at controlled opacity (e.g., 20–30%) to enhance feature overlap.
• Employs multiple diverse base images to create a set of poisons, preventing the network from learning distinct features for the target.
• In end-to-end training, leverages feature space collapse—where the target is pulled toward the base distribution—by using high poison diversity.
• Uses a leave-one-out ablation study to validate that optimization, diversity, and watermarking are all necessary for success.
• Employs influence functions and feature visualization to analyze why poisoning works differently in transfer learning vs. end-to-end training.

Experimental results

Research questions

• RQ1 Can targeted clean-label poisoning be achieved with minimal poisoning budget and no control over labeling?
• RQ2 Why is poisoning more effective in transfer learning than in end-to-end training, and how can this gap be closed?
• RQ3 How does watermarking with opacity control enhance the success of poisoning attacks in end-to-end training?
• RQ4 To what extent does poison diversity improve attack success in end-to-end training?
• RQ5 Can outlier targets (low-confidence examples) be more easily manipulated than standard targets?

Key findings

• The proposed clean-label attack achieves 100% success rate in the transfer learning scenario, outperforming prior work that achieved only 57% success on the same task.
• In end-to-end training, the attack achieves a success rate of up to 60% using approximately 50 poisoned images, with success increasing monotonically with the number of poisons.
• Targeting low-confidence (outlier) instances increases success rate to 70%, a 17% improvement over random targets.
• Watermarking with 30% opacity significantly improves success rates, while reducing opacity to 20% reduces performance, indicating sensitivity to feature overlap.
• The decision boundary remains nearly unchanged in end-to-end training, indicating that poisoning works via feature space drift rather than boundary rotation.
• Ablation studies confirm that all three components—optimization, diversity, and watermarking—are essential for successful poisoning in end-to-end training.