[Paper Review] Blackbox Attacks on Reinforcement Learning Agents Using Approximated Temporal Information

Yiren Zhao, Ilia Shumailov+4 more|arXiv (Cornell University)|Sep 6, 2019

Adversarial Robustness in Machine Learning45 references4 citations

TL;DR

This paper proposes a novel black-box attack on reinforcement learning (RL) agents using sequence-to-sequence models to predict future actions from observed observation sequences, enabling transferable adversarial perturbations without access to model internals. The key contribution is a time-bomb attack that triggers misbehavior after a delayed action, demonstrating a new class of stealthy, temporally targeted attacks on RL agents with over 80% action prediction accuracy.

ABSTRACT

Recent research on reinforcement learning (RL) has suggested that trained agents are vulnerable to maliciously crafted adversarial samples. In this work, we show how such samples can be generalised from White-box and Grey-box attacks to a strong Black-box case, where the attacker has no knowledge of the agents, their training parameters and their training methods. We use sequence-to-sequence models to predict a single action or a sequence of future actions that a trained agent will make. First, we show our approximation model, based on time-series information from the agent, consistently predicts RL agents' future actions with high accuracy in a Black-box setup on a wide range of games and RL algorithms. Second, we find that although adversarial samples are transferable from the target model to our RL agents, they often outperform random Gaussian noise only marginally. This highlights a serious methodological deficiency in previous work on such agents; random jamming should have been taken as the baseline for evaluation. Third, we propose a novel use for adversarial samplesin Black-box attacks of RL agents: they can be used to trigger a trained agent to misbehave after a specific time delay. This appears to be a genuinely new type of attack. It potentially enables an attacker to use devices controlled by RL agents as time bombs.

Motivation & Objective

• To develop a black-box attack on RL agents without access to model parameters or training details.
• To investigate whether adversarial samples can be generated and transferred in a fully black-box setting.
• To evaluate the effectiveness of adversarial attacks compared to random noise jamming as a baseline.
• To explore novel temporal attack surfaces, such as delayed misbehavior, in RL agents.
• To demonstrate a new class of time-bomb attacks that trigger future adversarial behavior after a delay.

Proposed method

• Train a sequence-to-sequence (seq2seq) model to predict future action sequences based solely on observed observation sequences from the target RL agent.
• Use the trained seq2seq model to generate adversarial perturbations on current observations that influence future actions.
• Apply the perturbations in a black-box setting where the attacker has no knowledge of the target agent’s architecture, hyperparameters, or training process.
• Use the l∞ norm constraint (e.g., ϵ = 0.3) to control perturbation magnitude and ensure imperceptibility.
• Evaluate attack success by measuring the drop in game score and the probability of triggering a specific delayed action.
• Compare attack performance against random Gaussian noise as a baseline to assess methodological robustness.

Experimental results

Research questions

• RQ1 Can adversarial samples be effectively generated and transferred in a fully black-box setting where no internal knowledge of the RL agent is available?
• RQ2 How does the performance of adversarial attacks compare to random Gaussian noise in reducing the game score of RL agents?
• RQ3 Can adversarial perturbations be crafted to induce misbehavior in a target RL agent with a specific time delay?
• RQ4 What factors influence the success rate of time-bomb attacks across different games and RL algorithms?
• RQ5 Is the seq2seq model’s action prediction capability sufficient to enable high-accuracy black-box attacks on diverse RL agents?

Key findings

• The seq2seq model achieved over 80% accuracy in predicting future action sequences across three games (CartPole, Pong, Space Invaders) and three different RL algorithms (DQN, A2C, Rainbow).
• Adversarial attacks based on the seq2seq model reduced the target agent’s game score, demonstrating transferability in a black-box setting.
• The performance of adversarial attacks was only marginally better than random Gaussian noise in reducing game scores, indicating a critical methodological flaw in prior work that lacked a proper noise baseline.
• The time-bomb attack successfully induced target agents to misbehave after a specific delay, with success rates exceeding 70% under larger attack budgets (ϵ > 0.7).
• The attack was more effective on A2C agents than on Rainbow agents, and more effective in Space Invaders than in Pong, suggesting game complexity and dynamics affect attack feasibility.
• The study reveals that the temporal dimension of RL agents is underexplored in adversarial RL, and time-bomb attacks represent a novel, stealthy attack vector with real-world implications for safety-critical systems.