[Paper Review] Data Poisoning Attacks in Contextual Bandits

Yuzhe Ma, Kwang-Sung Jun+2 more|arXiv (Cornell University)|Aug 17, 2018

Advanced Bandit Algorithms Research16 references4 citations

TL;DR

This paper proposes an optimization-based data poisoning attack framework for contextual bandits that manipulates historical rewards to force a target arm to be selected under a specific context. By solving a convex optimization problem, the attacker achieves high success rates with minimal side effects, especially in high-dimensional spaces, demonstrating a critical security vulnerability in reinforcement learning systems used in recommendation and healthcare applications.

ABSTRACT

We study offline data poisoning attacks in contextual bandits, a class of reinforcement learning problems with important applications in online recommendation and adaptive medical treatment, among others. We provide a general attack framework based on convex optimization and show that by slightly manipulating rewards in the data, an attacker can force the bandit algorithm to pull a target arm for a target contextual vector. The target arm and target contextual vector are both chosen by the attacker. That is, the attacker can hijack the behavior of a contextual bandit. We also investigate the feasibility and the side effects of such attacks, and identify future directions for defense. Experiments on both synthetic and real-world data demonstrate the efficiency of the attack algorithm.

Motivation & Objective

• To investigate the feasibility of offline data poisoning attacks in contextual bandit systems, which are widely used in online recommendation and adaptive medical treatment.
• To develop a general attack framework that manipulates historical rewards to force a specific arm to be selected under a target context.
• To analyze the side effects and detectability of such attacks, particularly in high-dimensional and real-world settings.
• To identify future research directions for both attack and defense mechanisms in contextual bandit systems.

Proposed method

• The attack framework formulates the poisoning problem as a convex optimization task, where the attacker modifies historical rewards to shift the estimated parameter vector toward favoring the target arm for the target context.
• The method uses a constrained optimization approach to ensure the perturbed rewards are within a small margin of the original values, minimizing detectability.
• The attacker specifies a target context $x^*$ and target arm $a^*$, and solves for reward perturbations that make the bandit algorithm's policy select $a^*$ when $x^*$ is observed.
• The optimization accounts for the regularization and noise model used in linear contextual bandits, ensuring the attack remains effective under standard learning algorithms.
• Side effects are quantified by measuring the fraction of contexts where the bandit policy deviates from the original due to the attack, using a test set of contexts.
• Experiments are conducted on both synthetic data and real-world datasets to evaluate attack success and side effect under varying dimensions, number of arms, and user frequencies.

Experimental results

Research questions

• RQ1 Can an attacker successfully hijack a contextual bandit policy by poisoning only the historical rewards, without altering the contexts or actions?
• RQ2 How does the side effect of the attack—defined as the fraction of contexts where the policy deviates—vary with the number of arms and problem dimension?
• RQ3 What is the detectability of the attack in high-dimensional settings, and how does it compare to low-dimensional or real-world user frequency scenarios?
• RQ4 Can the attack be extended to target multiple contexts simultaneously, and what are the challenges in doing so?
• RQ5 How can defenders leverage side effects to detect such poisoning attacks, and what strategies might attackers use to minimize detection?

Key findings

• The attack successfully forces the contextual bandit to select the target arm for the target context with high accuracy, even under realistic noise and regularization settings.
• In high-dimensional spaces (e.g., $d=200$), the side effect fraction drops to 0.035, indicating that attacks become significantly harder to detect as dimensionality increases.
• For real-world data, the side effect varied by user frequency: 53.91% and 50.40% for high-frequency users, but only 7.50% for a medium-frequency user, showing variable detectability.
• The median side effect across different numbers of arms ($K=2, 20, 200$) remained stable around 0.2–0.3, indicating no significant growth with $K$.
• The attack remains effective even when the target arm is the worst-performing arm under the target context, demonstrating strong control over the learning process.
• The results expose a critical security threat in real-world AI systems relying on contextual bandits, especially in applications like recommendation and healthcare where data poisoning can have long-term consequences.