[Paper Review] Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum

Junlin Wu, Yevgeniy Vorobeychik|arXiv (Cornell University)|Jun 21, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Bootstrapped Opportunistic Curriculum Learning (BCL), a novel adversarial curriculum framework that enhances robustness in deep reinforcement learning by bootstrapping each curriculum phase with the best-performing model from multiple prior runs and opportunistically skipping to higher perturbation levels when robustness is already achieved. BCL achieves state-of-the-art robustness, enabling policies to withstand adversarial perturbations up to ε=25/255 in Pong—significantly outperforming prior methods that fail beyond ε=5/255.

ABSTRACT

Despite considerable advances in deep reinforcement learning, it has been shown to be highly vulnerable to adversarial perturbations to state observations. Recent efforts that have attempted to improve adversarial robustness of reinforcement learning can nevertheless tolerate only very small perturbations, and remain fragile as perturbation size increases. We propose Bootstrapped Opportunistic Adversarial Curriculum Learning (BCL), a novel flexible adversarial curriculum learning framework for robust reinforcement learning. Our framework combines two ideas: conservatively bootstrapping each curriculum phase with highest quality solutions obtained from multiple runs of the previous phase, and opportunistically skipping forward in the curriculum. In our experiments we show that the proposed BCL framework enables dramatic improvements in robustness of learned policies to adversarial perturbations. The greatest improvement is for Pong, where our framework yields robustness to perturbations of up to 25/255; in contrast, the best existing approach can only tolerate adversarial noise up to 5/255. Our code is available at: https://github.com/jlwu002/BCL.

Motivation & Objective

• To address the critical vulnerability of deep reinforcement learning (DRL) policies to small adversarial perturbations in state observations.
• To overcome the limitations of existing adversarial training and naive curriculum approaches, which fail to scale robustness to larger perturbations.
• To develop a flexible, adaptive curriculum framework that improves robustness without sacrificing nominal performance.
• To enable DRL agents to generalize under increasingly severe adversarial attacks through structured, phase-wise curriculum progression.

Proposed method

• The framework employs a bootstrapped curriculum phase where the best-performing model from multiple adversarial training runs of the previous phase is selected as the starting point for the next.
• It introduces an opportunistic skip mechanism that advances the curriculum to higher perturbation levels (ε) if the current model demonstrates robustness to those levels.
• Adversarial examples are generated using a hybrid approach combining interval bound propagation and FGSM-based attacks to guide curriculum progression.
• The curriculum is structured to progressively increase the magnitude of adversarial perturbations, starting from small ε and advancing only when robustness is confirmed.
• Multiple training runs per phase ensure high-quality initialization and reduce variance in model performance.
• The method is evaluated on Atari environments (Pong, Freeway, BankHeist, RoadRunner) using DQN-style architectures with minimal impact on nominal reward.

Experimental results

Research questions

• RQ1 Can a curriculum learning framework significantly improve the robustness of DRL policies to larger adversarial perturbations compared to standard adversarial training?
• RQ2 Does bootstrapping each curriculum phase with the best model from multiple runs enhance robustness and training stability?
• RQ3 Can opportunistic skipping forward in the curriculum accelerate training while maintaining or improving robustness?
• RQ4 How does the proposed BCL framework compare to state-of-the-art methods like RADIAL-DQN and SA-DQN in terms of robustness under increasing perturbation magnitudes?
• RQ5 To what extent can the BCL framework maintain high nominal performance while achieving superior robustness across diverse Atari environments?

Key findings

• In Pong, BCL-trained policies achieve robust performance under adversarial perturbations up to ε=25/255, a dramatic improvement over the prior state of the art (RADIAL-DQN), which fails at ε=5/255.
• For BankHeist, BCL achieves an order-of-magnitude higher robustness at ε=15/255 compared to SA-DQN, demonstrating superior generalization under attack.
• In RoadRunner, BCL outperforms RADIAL-DQN by several orders of magnitude at ε=15/255, indicating substantial gains in robust policy learning.
• The BCL framework maintains strong nominal performance across all environments, with minimal degradation compared to standard DQN training.
• The opportunistic skip mechanism reduces training time by avoiding unnecessary phases, while the bootstrapped initialization ensures consistent performance gains across runs.
• The method shows consistent improvements across multiple runs and environments, with low variance in robustness metrics, indicating high reliability.