[Paper Review] Online Robustness Training for Deep Reinforcement Learning

M. L. Fischer, Matthew Mirman+2 more|arXiv (Cornell University)|Nov 3, 2019

Adversarial Robustness in Machine Learning44 references12 citations

TL;DR

This paper proposes RS-DQN, a novel deep reinforcement learning framework that decouples Q-network training from policy network (student) training to enable online robustness training. By distilling Q-values into a student network, RS-DQN integrates state-of-the-art adversarial and provably robust training methods, achieving strong robustness against gradient-based attacks—such as PGD—on Atari games while maintaining performance comparable to standard DQN.

ABSTRACT

In deep reinforcement learning (RL), adversarial attacks can trick an agent into unwanted states and disrupt training. We propose a system called Robust Student-DQN (RS-DQN), which permits online robustness training alongside Q networks, while preserving competitive performance. We show that RS-DQN can be combined with (i) state-of-the-art adversarial training and (ii) provably robust training to obtain an agent that is resilient to strong attacks during training and evaluation.

Motivation & Objective

• To address the vulnerability of deep reinforcement learning agents to gradient-based adversarial attacks that perturb input states and degrade performance.
• To enable the integration of advanced robust training techniques—previously used in supervised learning—into the deep RL training pipeline.
• To maintain competitive performance under clean (non-attacked) conditions while significantly improving robustness under adversarial perturbations.
• To demonstrate provable robustness to pixel-level perturbations (e.g., ±1 pixel) in Atari environments using symbolic robust training methods.
• To design a training framework that allows defensive training on the policy network without degrading the Q-network’s performance or learning dynamics.

Proposed method

• Decomposes standard DQN into a Q-network (for value estimation) and a student policy network (for action selection), enabling separate training of the policy with robustness objectives.
• Uses knowledge distillation to transfer Q-value predictions from the Q-network to the student network, ensuring the student learns the optimal policy.
• Applies adversarial training to the student network using PGD-based adversarial states that minimize the probability of selecting the optimal action.
• Employs provably robust training via symbolic interval analysis (using DiffAI) to guarantee correct action selection within a bounded perturbation region around each state.
• Gradually anneals the robustness loss weight and perturbation radius during training to balance accuracy and robustness.
• Maintains standard DQN training for the Q-network while using the student network for exploration and action selection, preserving training stability.

Experimental results

Research questions

• RQ1 Can state-of-the-art adversarial training methods from supervised learning be effectively transferred to deep reinforcement learning for robust policy learning?
• RQ2 Can provably robust training be integrated into the DQN framework to guarantee robustness against bounded input perturbations in RL environments?
• RQ3 Does decoupling policy learning from value network training allow for robust training without degrading the performance of the Q-network?
• RQ4 To what extent can a DQN agent maintain performance under clean conditions while achieving strong robustness under adversarial attacks?
• RQ5 What is the maximum perturbation radius (in pixel intensity) for which the agent’s action remains provably correct in Atari games?

Key findings

• RS-DQN achieves performance comparable to standard DQN under clean evaluation conditions, with average scores of 32.53 (Freeway), 154.00 (Bank Heist), and 90.47 (Boxing), slightly below DQN’s 33.00, 222.00, and 95.87 respectively.
• Under PGD attacks (k=4), DQN’s performance collapses (e.g., 5.13 in Pong), while RS-DQN maintains robustness, scoring 5.13 in Pong and 90.47 in Boxing.
• With provably robust training, RS-DQN achieves a minimum provable robustness radius of ε_max = 2.028 (multiplied by 255), meaning it is certifiably robust to ±1 pixel changes in Freeway.
• In Bank Heist, RS-DQN achieves ε_max = 1.373, indicating robustness to ±1 pixel perturbations with guaranteed action correctness.
• The provably robust RS-DQN agent maintains a significant performance gap compared to DQN (154.00 vs. 222.00 in Bank Heist), but this trade-off is acceptable given the strong robustness guarantee.
• The method successfully enables online robustness training without degrading the Q-network’s learning, as shown by stable training and evaluation curves across all Atari environments tested.