[Paper Review] Data Poisoning Attacks against Online Learning

Yizhen Wang, Kamalika Chaudhuri|arXiv (Cornell University)|Aug 27, 2018

Adversarial Robustness in Machine Learning2 references55 citations

TL;DR

This paper formalizes data poisoning attacks for online learning, proposing optimization-based strategies to poison streaming data and evaluating their effectiveness across semi-online and fully-online settings.

ABSTRACT

We consider data poisoning attacks, a class of adversarial attacks on machine learning where an adversary has the power to alter a small fraction of the training data in order to make the trained classifier satisfy certain objectives. While there has been much prior work on data poisoning, most of it is in the offline setting, and attacks for online learning, where training data arrives in a streaming manner, are not well understood. In this work, we initiate a systematic investigation of data poisoning attacks for online learning. We formalize the problem into two settings, and we propose a general attack strategy, formulated as an optimization problem, that applies to both with some modifications. We propose three solution strategies, and perform extensive experimental evaluation. Finally, we discuss the implications of our findings for building successful defenses.

Motivation & Objective

• Motivate studying data poisoning in online learning settings where data arrives as a stream.
• Formalize semi-online and fully-online attack settings and define attacker objectives.
• Develop optimization-based attack strategies that adapt to online gradient descent updates.
• Analyze how data order and learning rates affect attack efficacy.
• Discuss defense implications and potential robust online learning approaches.

Proposed method

• Model online learning as online gradient descent with streaming data and a convex loss plus regularizer.
• Formulate attacker optimization to modify at most K points in the stream under a feasible set F.
• Introduce three attack paradigms: Incremental Attack, Interval Attack, and Teach-and-Reinforce Attack.
• Apply label inversion as needed and smooth objective functions to enable gradient-based optimization.
• Compute gradients via chain rule with a recurrent prefix formulation to handle online updates.
• Evaluate attacks on four datasets under semi-online and fully-online settings with various learning-rate regimes.

Experimental results

Research questions

• RQ1 Can an adversary effectively poison data in online learning by modifying a small fraction of the data stream?
• RQ2 How does the online setting (semi-online vs fully-online) and learning rate affect the strength of data poisoning attacks?
• RQ3 Where in the data stream are attack modifications most impactful under different settings?
• RQ4 Do gradient-based online attacks outperform oblivious or label-flip baselines in online learning?
• RQ5 What defenses might mitigate online data poisoning threats in streaming classifiers?

Key findings

• Online adversaries significantly outperform oblivious or label-flip baselines in poisoned online learning.
• Attack efficacy depends on learning rate; fast-decaying rates are more susceptible to poisoning.
• Semi-online settings tend to be more vulnerable than fully-online settings to these attacks.
• Incremental and Interval attacks are generally more effective in semi-online, while Teach-and-Reinforce excels in fully-online scenarios.
• Attack position patterns differ by dataset and setting, with end-of-stream emphasis in some cases and beginning-of-stream emphasis in others.
• Gradient-based online attacks remain effective across datasets such as synthetic, MNIST, and Spambase, highlighting the online nature as a key vulnerability.