[Paper Review] Manipulating Reinforcement Learning: Poisoning Attacks on Cost Signals

Yunhan Huang, Quanyan Zhu|arXiv (Cornell University)|Feb 7, 2020

Adversarial Robustness in Machine Learning23 references4 citations

TL;DR

This paper investigates stealthy poisoning attacks on reinforcement learning by manipulating cost signals, showing that TD-learning's approximation error is bounded by a constant multiple of attack magnitude, independent of the λ parameter. For Q-learning, it identifies a robust region where attacks cannot alter the learned policy, establishing fundamental limits for both offensive and defensive strategies in adversarial RL settings.

ABSTRACT

This chapter studies emerging cyber-attacks on reinforcement learning (RL) and introduces a quantitative approach to analyze the vulnerabilities of RL. Focusing on adversarial manipulation on the cost signals, we analyze the performance degradation of TD($λ$) and $Q$-learning algorithms under the manipulation. For TD($λ$), the approximation learned from the manipulated costs has an approximation error bound proportional to the magnitude of the attack. The effect of the adversarial attacks on the bound does not depend on the choice of $λ$. In $Q$-learning, we show that $Q$-learning algorithms converge under stealthy attacks and bounded falsifications on cost signals. We characterize the relation between the falsified cost and the $Q$-factors as well as the policy learned by the learning agent which provides fundamental limits for feasible offensive and defensive moves. We propose a robust region in terms of the cost within which the adversary can never achieve the targeted policy. We provide conditions on the falsified cost which can mislead the agent to learn an adversary's favored policy. A case study of TD($λ$) learning is provided to corroborate the results.

Motivation & Objective

• To analyze the vulnerability of reinforcement learning (RL) algorithms to adversarial manipulation of cost signals.
• To quantify the performance degradation of TD-learning and Q-learning under stealthy cost signal falsifications.
• To establish fundamental limits on adversarial attacks by characterizing the robust region in Q-learning where attacks cannot achieve their objective.
• To provide theoretical bounds on approximation error induced by cost signal manipulation in TD-learning.
• To offer insights for designing robust RL systems resilient to deceptive feedback attacks.

Proposed method

• Formulates a general attack model where an adversary manipulates cost signals in a Markov decision process framework.
• Analyzes TD(λ) learning under manipulated costs, deriving a bound on approximation error proportional to the attack magnitude, independent of λ.
• Applies the TD(λ) algorithm with function approximation using a parameterized value function to estimate cost-to-go.
• Uses a case study with a 20-state Markov chain problem to simulate and validate theoretical bounds on error propagation.
• Characterizes the relationship between falsified costs and learned Q-factors in Q-learning, identifying conditions under which attacks can mislead the agent.
• Derives a robust region in cost space where the adversary cannot induce a targeted policy, enabling strategic assessment for both attackers and defenders.

Experimental results

Research questions

• RQ1 How does adversarial manipulation of cost signals affect the convergence and accuracy of TD-learning algorithms?
• RQ2 What is the fundamental bound on the approximation error in TD(λ) learning when cost signals are falsified?
• RQ3 Under what conditions can an adversary successfully mislead a Q-learning agent into adopting a desired policy through cost signal manipulation?
• RQ4 What is the size and structure of the robust region in cost space where attacks cannot alter the learned policy in Q-learning?
• RQ5 How does the choice of λ in TD(λ) influence the sensitivity of the learning algorithm to cost signal manipulation?

Key findings

• The approximation error in TD(λ) learning under cost signal manipulation is bounded by a constant times the magnitude of the attack, and this bound is independent of the λ parameter.
• In Q-learning, the adversary cannot achieve its objective if the falsified cost lies within a defined robust region, which is characterized by specific constraints on the cost deviation.
• The theoretical bound on the approximation error in TD(λ) learning is validated through a case study using a 20-state Markov chain, showing consistent error growth proportional to the attack magnitude.
• Even a single-point manipulation of the cost signal at state i20 causes significant deviation in the estimated cost-to-go function across all states, demonstrating error propagation.
• The robust region in Q-learning provides a strategic boundary: defenders can assess system resilience, and attackers can evaluate the feasibility of their goals.
• The results demonstrate that while attacks can degrade performance, there are inherent limits to their effectiveness, especially in Q-learning where a safe region of costs exists.