[Paper Review] Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning

Chao-Han Huck Yang, Jun Qi+5 more|arXiv (Cornell University)|Feb 20, 2020

Adversarial Robustness in Machine Learning24 references25 citations

TL;DR

This paper proposes enhanced adversarial strategically-timed attacks against deep reinforcement learning (DRL) navigation systems by injecting physical noise at strategically selected time frames. It introduces a white-box attack using weighted-majority online learning for dynamic timing decisions and a black-box attack via policy gradient-based imitation, both significantly degrading DRL performance across three 3D environments (robot arm control, banana collection, self-driving car) under zero-out, Gaussian, and FGSM adversarial noise patterns.

ABSTRACT

Recent deep neural networks based techniques, especially those equipped with the ability of self-adaptation in the system level such as deep reinforcement learning (DRL), are shown to possess many advantages of optimizing robot learning systems (e.g., autonomous navigation and continuous robot arm control.) However, the learning-based systems and the associated models may be threatened by the risks of intentionally adaptive (e.g., noisy sensor confusion) and adversarial perturbations from real-world scenarios. In this paper, we introduce timing-based adversarial strategies against a DRL-based navigation system by jamming in physical noise patterns on the selected time frames. To study the vulnerability of learning-based navigation systems, we propose two adversarial agent models: one refers to online learning; another one is based on evolutionary learning. Besides, three open-source robot learning and navigation control environments are employed to study the vulnerability under adversarial timing attacks. Our experimental results show that the adversarial timing attacks can lead to a significant performance drop, and also suggest the necessity of enhancing the robustness of robot learning systems.

Motivation & Objective

• To investigate the vulnerability of DRL-based navigation systems to timing-based adversarial attacks.
• To develop a white-box attack strategy using weighted-majority online learning for optimal attack timing.
• To design a black-box attack method that imitates the victim's policy without model access.
• To evaluate the robustness of DRL agents under various noise patterns (zero-out, Gaussian, FGSM) in realistic 3D environments.
• To demonstrate that both value-based (DQN) and policy-based (A3C) DRL models are highly susceptible to such attacks.

Proposed method

• Proposes a noisy observation framework: noisy_st = st + noise(t), where noise(t) is injected at selected time steps.
• Defines three noise types: T1 (pulsed zero-out), T2 (Gaussian sensor fusion noise), and T3 (FGSM-generated adversarial patterns with ℓ∞-norm constraint).
• Develops a white-box attack using weighted-majority algorithm (WMA) to estimate potential energy and select attack times based on action cost differences.
• Introduces a black-box attack via PEPG-ASA, where an attacker trains a policy agent using inverse reinforcement learning to mimic the victim's behavior.
• Uses gradient estimation via log-probability sampling to optimize attack policy: ∇ρE(ρ) ≈ 1/N Σ ∇ρ log p(θ|ρ)r(hn).
• Employs three open-source Unity-3D environments: Env1 (continuous robot arm), Env2 (banana collector), Env3 (Donkey Car self-driving simulation).

Experimental results

Research questions

• RQ1 Can strategically-timed adversarial attacks significantly degrade the performance of DRL-based navigation systems?
• RQ2 How effective is a white-box attack using online learning with weighted-majority for timing selection compared to baseline random or prior methods?
• RQ3 Can a black-box attacker successfully compromise DRL agents without access to model parameters by learning from actions and rewards?
• RQ4 How do different noise patterns (zero-out, Gaussian, FGSM) affect the robustness of DRL agents in continuous control and navigation tasks?
• RQ5 To what extent do both DQN and A3C agents fail under these timing-based attacks in realistic 3D environments?

Key findings

• The WMA white-box attack reduced average returns by up to 86% in Env1 (DQN) and 89% in Env2 (A3C), with rewards dropping from ~30 to ~3.
• The black-box PEPG-ASA attack achieved comparable degradation, reducing rewards from ~12 to ~5 in Env2 (DQN) and ~1.1 to ~0.6 in Env3 (A3C).
• Even under the same noise pattern (e.g., FGSM), timing-based attacks were significantly more effective than random jamming, which only reduced rewards by ~20%.
• The Lin et al. [16] baseline method showed similar performance to PEPG-ASA, but WMA outperformed it in both white-box and black-box settings.
• All attacks caused consistent performance drops across both DQN and A3C agents, indicating broad vulnerability in value-based and policy-based DRL.
• The results demonstrate that DRL agents are highly sensitive to adversarial timing, even when noise is limited to a few frames, highlighting the need for robust defense mechanisms.