[Paper Review] Certifiable Robustness to Adversarial State Uncertainty in Deep Reinforcement Learning

Michael Everett, Björn Lütjens+1 more|arXiv (Cornell University)|Apr 11, 2020

Adversarial Robustness in Machine LearningComputer Science72 references36 citations

TL;DR

This paper proposes CARRL, a certifiably robust deep reinforcement learning framework that uses robust optimization to compute guaranteed lower bounds on state-action values under worst-case input perturbations from sensor noise or adversarial attacks. By providing a certificate of solution quality even when the true state is unknown, CARRL enables safe action selection in safety-critical environments, demonstrating improved robustness in pedestrian collision avoidance and Atari Pong with provable performance guarantees.

ABSTRACT

Deep Neural Network-based systems are now the state-of-the-art in many robotics tasks, but their application in safety-critical domains remains dangerous without formal guarantees on network robustness. Small perturbations to sensor inputs (from noise or adversarial examples) are often enough to change network-based decisions, which was recently shown to cause an autonomous vehicle to swerve into another lane. In light of these dangers, numerous algorithms have been developed as defensive mechanisms from these adversarial inputs, some of which provide formal robustness guarantees or certificates. This work leverages research on certified adversarial robustness to develop an online certifiably robust for deep reinforcement learning algorithms. The proposed defense computes guaranteed lower bounds on state-action values during execution to identify and choose a robust action under a worst-case deviation in input space due to possible adversaries or noise. Moreover, the resulting policy comes with a certificate of solution quality, even though the true state and optimal action are unknown to the certifier due to the perturbations. The approach is demonstrated on a Deep Q-Network policy and is shown to increase robustness to noise and adversaries in pedestrian collision avoidance scenarios and a classic control task. This work extends one of our prior works with new performance guarantees, extensions to other RL algorithms, expanded results aggregated across more scenarios, an extension into scenarios with adversarial behavior, comparisons with a more computationally expensive method, and visualizations that provide intuition about the robustness algorithm.

Motivation & Objective

• Address the lack of formal robustness guarantees in deep RL policies, especially under sensor noise or adversarial input perturbations.
• Overcome the limitations of standard DRL agents that fail under small input perturbations, such as causing autonomous vehicles to swerve into lanes.
• Develop a method that provides formal, certifiable robustness to input uncertainty without requiring retraining or changing the underlying policy.
• Ensure the resulting policy comes with a verifiable certificate of solution quality, even when the true state and optimal action are unknown due to perturbations.
• Extend efficient robustness verification techniques from supervised learning to deep RL, adapting them to handle heterogeneous input scales common in control tasks.

Proposed method

• Formulate a robust optimization problem that selects the action maximizing the worst-case value over an ϵ-ball of possible state perturbations around the observed input.
• Use relaxed robustness verification techniques (e.g., Fast-Lin-style linear relaxations of ReLU activations) to efficiently compute guaranteed lower bounds on Q-values under input uncertainty.
• Introduce vector-valued ϵ to handle inputs with varying scales (e.g., position, angle, torque), enabling application to robotic control tasks.
• Derive a certificate of suboptimality by bounding the gap between the robust action’s value and the optimal value at the true (unknown) state, using the lower bound and nominal Q-value.
• Integrate the robust decision-making module as a post-processing layer on top of a pre-trained DQN, enabling deployment without retraining.
• Extend the framework to policy-based RL algorithms and scenarios involving adversarial agents, demonstrating generalization beyond image-based attacks.

Experimental results

Research questions

• RQ1 Can we provide formal, certifiable robustness guarantees for deep reinforcement learning agents under adversarial or noisy state observations?
• RQ2 How can we compute guaranteed lower bounds on Q-values under worst-case input perturbations while maintaining computational efficiency?
• RQ3 Can we extend robustness verification techniques from supervised learning to deep RL, particularly for environments with heterogeneous input scales?
• RQ4 How does the proposed method perform compared to non-certified baselines and more computationally expensive alternatives in terms of robustness and solution quality?
• RQ5 Can the framework be extended to scenarios involving adversarial behavior in the environment, such as competing agents or dynamic obstacles?

Key findings

• CARRL significantly reduces collision rates in pedestrian collision avoidance scenarios, demonstrating improved robustness under adversarial state perturbations.
• The method increases average episodic return in the CartPole control task under noisy observations, showing improved performance stability.
• In Atari Pong, the CARRL-enhanced DQN policy achieves higher rewards and greater robustness compared to the standard DQN under adversarial input attacks.
• The algorithm provides a verifiable certificate of suboptimality, quantifying how far the robust action is from the optimal action at the true state, even when the true state is unknown.
• Visualizations show that the robustness bounds become more conservative as ϵ increases, particularly due to degradation in ReLU linear approximations, but remain tight for small perturbations.
• Comparisons with a more computationally expensive robustness method show that CARRL achieves comparable robustness with significantly lower inference cost, validating its efficiency.