[Paper Review] Defending Adversarial Attacks without Adversarial Attacks in Deep Reinforcement Learning.

Xinghua Qu, Yew-Soon Ong+2 more|arXiv (Cornell University)|Aug 14, 2020

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Robust Policy Distillation, a novel distillation method that enhances adversarial robustness in deep reinforcement learning without requiring adversarial examples during student policy training. By combining prescription gap maximization and Jacobian regularization losses, it achieves state-of-the-art robustness on five Atari games, demonstrating both theoretical guarantees and empirical superiority over existing methods.

ABSTRACT

Many recent studies in deep reinforcement learning (DRL) have proposed to boost adversarial robustness through policy distillation utilizing adversarial training, where additional adversarial examples are added in the training process of the student policy; this makes the robustness improvement less flexible and more computationally expensive. In contrast, we propose an efficient policy distillation paradigm called robust policy distillation that is capable of achieving an adversarially robust student policy without relying on any adversarial example during student policy training. To this end, we devise a new policy distillation loss that consists of two terms: 1) a prescription gap maximization loss aiming at simultaneously maximizing the likelihood of the action selected by the teacher policy and the entropy over the remaining actions; 2) a Jacobian regularization loss that minimizes the magnitude of Jacobian with respect to the input state. The theoretical analysis proves that our distillation loss guarantees to increase the prescription gap and the adversarial robustness. Meanwhile, experiments on five Atari games firmly verifies the superiority of our policy distillation on boosting adversarial robustness compared to other state-of-the-arts.

Motivation & Objective

• Address the computational inefficiency and rigidity of existing adversarial training-based distillation methods in deep reinforcement learning.
• Overcome the dependency on adversarial examples during student policy training, which limits flexibility and increases training cost.
• Develop a distillation framework that ensures adversarial robustness through intrinsic loss design rather than data augmentation with adversarial examples.
• Theoretically justify the robustness improvement via a new loss function that maximizes prescription gap and minimizes input sensitivity.
• Demonstrate empirical superiority of the proposed method on standard DRL benchmarks, specifically five Atari games, under adversarial attacks.

Proposed method

• Propose a new policy distillation loss combining two components: prescription gap maximization and Jacobian regularization.
• Design the prescription gap maximization loss to increase the probability of the teacher’s action while promoting entropy over other actions, enhancing robustness to input perturbations.
• Introduce the Jacobian regularization loss to minimize the magnitude of the policy’s Jacobian with respect to the input state, reducing sensitivity to small input changes.
• Theoretical analysis proves that the combined loss function increases the prescription gap and improves adversarial robustness under bounded perturbations.
• Train the student policy using only clean data, avoiding any adversarial examples during training, while still achieving robust performance.
• The method relies solely on the teacher’s behavior and gradient information, making it efficient and scalable without adversarial data generation.

Experimental results

Research questions

• RQ1 Can adversarial robustness in deep reinforcement learning be achieved without using adversarial examples during student policy training?
• RQ2 Does a distillation loss that maximizes prescription gap and regularizes the policy’s Jacobian improve robustness against adversarial attacks?
• RQ3 How does the proposed method compare to state-of-the-art adversarial training and distillation approaches in terms of robustness and training efficiency?
• RQ4 What theoretical guarantees does the proposed loss function provide regarding adversarial robustness and prescription gap maximization?
• RQ5 Can the method generalize across diverse environments, as demonstrated on a diverse set of Atari games?

Key findings

• The proposed Robust Policy Distillation method achieves state-of-the-art adversarial robustness on five Atari games without using any adversarial examples during student training.
• The method demonstrates significant improvement in robustness compared to existing state-of-the-art approaches, as validated through comprehensive evaluation under adversarial attacks.
• Theoretical analysis confirms that the proposed loss function increases the prescription gap and enhances adversarial robustness by minimizing input sensitivity via Jacobian regularization.
• The method maintains high performance on clean environments while significantly improving robustness, indicating no trade-off between standard performance and robustness.
• Experiments show that the distillation process is efficient and scalable, avoiding the computational burden of generating adversarial examples during training.
• The ablation study confirms that both components of the loss—prescription gap maximization and Jacobian regularization—are essential for achieving robust performance.