[Paper Review] Certified Adversarial Robustness for Deep Reinforcement Learning

Björn Lütjens, Michael Everett+1 more|arXiv (Cornell University)|Oct 28, 2019

Adversarial Robustness in Machine Learning49 references46 citations

TL;DR

The paper introduces an online certified defense (CARRL) that computes guaranteed lower bounds on state-action values under bounded input perturbations to select actions robust to adversaries and noise in deep RL tasks.

ABSTRACT

Deep Neural Network-based systems are now the state-of-the-art in many robotics tasks, but their application in safety-critical domains remains dangerous without formal guarantees on network robustness. Small perturbations to sensor inputs (from noise or adversarial examples) are often enough to change network-based decisions, which was already shown to cause an autonomous vehicle to swerve into oncoming traffic. In light of these dangers, numerous algorithms have been developed as defensive mechanisms from these adversarial inputs, some of which provide formal robustness guarantees or certificates. This work leverages research on certified adversarial robustness to develop an online certified defense for deep reinforcement learning algorithms. The proposed defense computes guaranteed lower bounds on state-action values during execution to identify and choose the optimal action under a worst-case deviation in input space due to possible adversaries or noise. The approach is demonstrated on a Deep Q-Network policy and is shown to increase robustness to noise and adversaries in pedestrian collision avoidance scenarios and a classic control task.

Motivation & Objective

• Motivate the need for robustness guarantees in deep RL for safety-critical robotics tasks.
• Adapt and extend robustness certification methods to deep RL with variable-scale inputs.
• Provide an online defense that selects actions using worst-case Q-value bounds.
• Demonstrate robustness improvements in collision avoidance and cartpole domains.

Proposed method

• Define a certified lower bound Q_L(s_adv, a_j) as the minimum of Q_l(s, a_j) over s in an epsilon-ball around s_adv.
• Extend robustness certification to vector-valued epsilon to handle varied input scales.
• Derive a closed-form computation for Q_L using the network’s A matrices and ReLU bound parameters.
• Integrate Q_L into action selection by choosing a* = argmax_j Q_L(s_adv, a_j).
• Apply the approach to Deep Q-Networks (DQN) with an online defense node during execution.
• Demonstrate real-time feasibility (forward pass ~2ms for bounds, ~20ms total per action query).

Experimental results

Research questions

• RQ1 How can robustness certification be adapted to deep RL to provide guarantees under input perturbations?
• RQ2 Can vector-valued input perturbations (different sensor scales) be handled effectively in certification bounds?
• RQ3 Does using certified lower bounds on Q-values improve safety and performance under adversarial or noisy observations in RL tasks?
• RQ4 What is the impact of the certified defense on performance in collision avoidance and classic control tasks?
• RQ5 Is the certified defense feasible for online execution in real-time RL systems?

Key findings

• The proposed certified defense (CARRL) reduces collisions under adversarial perturbations and noise in collision avoidance scenarios.
• The approach yields robust action selection by maximizing Q_L, improving rewards under perturbations while keeping nominal performance under no perturbation.
• Extending certification to vector-valued perturbations enables handling varied sensor scales in robotics observations.
• The method generalizes to cartpole, showing improved resilience to noise with a modest loss in some settings.
• Runtime results indicate the defense is viable for real-time use (forward pass times and overall latency reported).