[Paper Review] Adaptive Reward-Poisoning Attacks against Reinforcement Learning

Xuezhou Zhang, Yuzhe Ma+2 more|arXiv (Cornell University)|Mar 27, 2020

Adversarial Robustness in Machine Learning31 references34 citations

TL;DR

The paper formalizes reward-poisoning attacks against RL, proves when such attacks are feasible or infeasible under L-infinity perturbations, and introduces fast adaptive attacks that achieve target policies in polynomial time, with empirical DRL-based attack methods.

ABSTRACT

In reward-poisoning attacks against reinforcement learning (RL), an attacker can perturb the environment reward $r_t$ into $r_t+δ_t$ at each step, with the goal of forcing the RL agent to learn a nefarious policy. We categorize such attacks by the infinity-norm constraint on $δ_t$: We provide a lower threshold below which reward-poisoning attack is infeasible and RL is certified to be safe; we provide a corresponding upper threshold above which the attack is feasible. Feasible attacks can be further categorized as non-adaptive where $δ_t$ depends only on $(s_t,a_t, s_{t+1})$, or adaptive where $δ_t$ depends further on the RL agent's learning process at time $t$. Non-adaptive attacks have been the focus of prior works. However, we show that under mild conditions, adaptive attacks can achieve the nefarious policy in steps polynomial in state-space size $|S|$, whereas non-adaptive attacks require exponential steps. We provide a constructive proof that a Fast Adaptive Attack strategy achieves the polynomial rate. Finally, we show that empirically an attacker can find effective reward-poisoning attacks using state-of-the-art deep RL techniques.

Motivation & Objective

• Motivate the study of training-time reward poisoning in RL where rewards are perturbed by an attacker.
• Characterize feasibility regimes for reward-poisoning under an infinity-norm constraint on perturbations.
• Develop both non-adaptive and fast adaptive attack strategies and bound their costs.
• Provide empirical evidence that attackers can find effective reward-poisoning attacks using deep RL techniques.

Proposed method

• Model the attack as an inference between the environment MDP, the learning agent using Q-learning, and an attacker with bounded perturbations delta_t to rewards.
• Define a target policy set pi^dagger and the corresponding target Q-table set Q^dagger to formalize the attack objective.
• Prove boundedness results for Q-learning under attack and derive infeasibility certificates (Delta_1 and Delta_2) and feasibility certificates (Delta_3) for non-adaptive attacks.
• Introduce a Fast Adaptive Attack (FAA) that uses Q_t and navigates to target states, achieving polynomial-time attack costs under certain sparsity and Delta conditions.
• Present an adaptive attack policy phi^xi_FAA and analyze its cost bounds (Theorem 5) and corollaries for grid-like environments.
• Demonstrate empirically via RL experiments that adaptive attacks outperform non-adaptive ones and that DRL can discover effective attack policies (TD3-based).

Experimental results

Research questions

• RQ1 Under which L-infinity reward perturbation magnitudes Delta is reward poisoning feasible or infeasible for forcing a target policy?
• RQ2 How do non-adaptive versus adaptive attack strategies compare in terms of attack cost and time to enforce pi^dagger?
• RQ3 Can adaptive attacks achieve polynomial-time enforcement of target policies, and what structural assumptions (e.g., sparsity of target states) are required?
• RQ4 Can data-driven methods (e.g., deep RL) discover effective reward-poisoning strategies beyond analytically constructed attacks?
• RQ5 What robustness certificates can certify safe RL behavior under reward poisoning?

Key findings

• There exist Delta thresholds (Delta_1, Delta_2) below which the RL agent remains safe and eventually converges to the optimal policy.
• There exists a higher threshold Delta_3 above which a non-adaptive attack can feasibly enforce the target policy, with attack cost bounded by O(L^5) (L = covering number).
• A fast adaptive attack (FAA) achieves polynomial-time enforcement of the target policy for sparse target states, with cost bounds that scale poly approximately with state space size |S| and other factors.
• Experiments show adaptive attacks (FAA) can significantly outperform non-adaptive attacks in efficiency, and deep RL (TD3) can discover effective attack policies.
• Empirical results on chain MDPs and grid worlds illustrate polynomial-time attack costs for adaptive methods versus exponential scaling for non-adaptive methods.