[Paper Review] Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning

Amin Rakhsha, Goran Radanović+3 more|arXiv (Cornell University)|Mar 28, 2020

Adversarial Robustness in Machine Learning39 citations

TL;DR

The paper develops a framework for training-time data poisoning in reinforcement learning to force an agent to adopt a targeted policy, assessing offline planning and online learning threats under reward and transition-dynamics poisoning.

ABSTRACT

We study a security threat to reinforcement learning where an attacker poisons the learning environment to force the agent into executing a target policy chosen by the attacker. As a victim, we consider RL agents whose objective is to find a policy that maximizes average reward in undiscounted infinite-horizon problem settings. The attacker can manipulate the rewards or the transition dynamics in the learning environment at training-time and is interested in doing so in a stealthy manner. We propose an optimization framework for finding an \\emph{optimal stealthy attack} for different measures of attack cost. We provide sufficient technical conditions under which the attack is feasible and provide lower/upper bounds on the attack cost. We instantiate our attacks in two settings: (i) an \\emph{offline} setting where the agent is doing planning in the poisoned environment, and (ii) an \\emph{online} setting where the agent is learning a policy using a regret-minimization framework with poisoned feedback. Our results show that the attacker can easily succeed in teaching any target policy to the victim under mild conditions and highlight a significant security threat to reinforcement learning agents in practice.

Motivation & Objective

• Motivate and formalize training-time poisoning threats against RL agents operating under average reward in undiscounted infinite-horizon MDPs.
• Develop optimization-based methods to construct stealthy poisoning strategies that make a target policy ε-robust optimal.
• Differentiate attack models for offline planning and online learning, and for rewards versus transition dynamics.
• Provide theoretical bounds on attack feasibility and cost, plus constructive approaches and simulations.

Proposed method

• Model RL as an average-reward MDP and define ε-robust optimality for a target policy.
• Formulate poisoning problems that modify rewards or transition dynamics to make the target policy optimal (offline) or to steer online learners toward the target policy.
• Introduce neighbor-policy based verification (π{s;a}) to reduce constraint complexity from exponential to linear in S and A.
• Derive convex/structured optimization problems (P1, P3) and non-convex formulations (P2, P4) for poisoning rewards or transitions and analyze feasibility.
• Provide sufficient conditions and bounds on attack costs using Hajnal measure (α) and χ̄ terms, and relate offline results to online sampling-based attacks.

Experimental results

Research questions

• RQ1 Can an attacker poison the environment to make a chosen target policy ε-robust optimal in offline planning?
• RQ2 How can rewards or transition dynamics be poisoned to stealthily steer online RL learners toward a target policy while minimizing attack cost?
• RQ3 What are the theoretical bounds on the cost of such poisoning attacks and the conditions that guarantee feasibility?
• RQ4 How do offline poisoning results translate into practical online attack templates using sampling MDPs?
• RQ5 What are the differences in attack design between poisoning rewards versus poisoning dynamics?

Key findings

• An attacker can feasibly and stealthily teach a target policy under mild conditions, with quantifiable cost bounds.
• Offline planning attacks via reward poisoning yield convex optimization (P1) with feasible solutions and bounds linking cost to χ̄ and α.
• Offline dynamics poisoning attacks lead to (generally non-convex) problems (P2) but provide sufficient conditions and upper/lower bounds for attack cost.
• Online attacks leverage a sampling MDP approach, tying average mismatch and average cost to the regret of the learner and to χ̄ε, guiding attack design.
• For online attacks, the framework shows that o(1) average mismatch is achievable when the sampling MDP is ergodic and the target policy is ε-robust optimal.