[Paper Review] Safe Reinforcement Learning via Curriculum Induction

Matteo Turchetta, Andrey Kolobov+3 more|arXiv (Cornell University)|Jun 22, 2020

Reinforcement Learning in RoboticsComputer Science44 references41 citations

TL;DR

CISR introduces a curriculum-based safe RL framework where a teaching agent uses interventions to keep the learner safe during training, while evolving curricula across generations to improve final policy performance.

ABSTRACT

In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. In such settings, the agent needs to behave safely not only after but also while learning. To achieve this, existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations during exploration with high probability, but both the probabilistic guarantees and the smoothness assumptions inherent in the priors are not viable in many scenarios of interest such as autonomous driving. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor that saves the agent from violating constraints during learning. In this model, we introduce the monitor that neither needs to know how to do well at the task the agent is learning nor needs to know how the environment works. Instead, it has a library of reset controllers that it activates when the agent starts behaving dangerously, preventing it from doing damage. Crucially, the choices of which reset controller to apply in which situation affect the speed of agent learning. Based on observing agents' progress, the teacher itself learns a policy for choosing the reset controllers, a curriculum, to optimize the agent's final policy reward. Our experiments use this framework in two environments to induce curricula for safe and efficient learning.

Motivation & Objective

• Motivate safe RL in safety-critical settings where exploration can be costly or dangerous.
• Propose CISR, a teacher-student framework that uses interventions to ensure safety during learning without requiring environment models.
• Develop a curriculum policy that optimizes the sequence of interventions based on observed student progress.
• Provide theoretical guarantees showing safety properties of intervention-induced CMDPs.
• Demonstrate empirical safety and efficiency benefits in challenging environments and show Curriculum Policy transfer across agents.

Proposed method

• Define interventions as trigger sets with state-conditioned reset distributions.
• Model each intervention as an altered CMDP that preserves feasibility and can override dynamics to keep the learner safe.
• Introduce a student learning problem under interventions with constraints on safety violations and teacher-imposed constraints.
• Formalize curricula as sequences of intervention CMDPs and define a curriculum policy that adapts based on student performance statistics.
• Treat the teacher as an online learner that optimizes a curriculum policy across rounds using evaluation features and GP-UCB for parameter optimization.
• Describe practical implementation choices including CMDP solvers with primal-dual optimization, knowledge transfer across interventions, and a reactive teacher policy with a Bayesian optimization loop.

Experimental results

Research questions

• RQ1 How can a teacher guarantee safety of an RL agent during learning without full knowledge of the task or environment?
• RQ2 Can a data-driven, adaptive curriculum policy accelerate safe learning over fixed or no curricula?
• RQ3 Do intervention-induced CMDPs yield safe learning that transfers across students and tasks?
• RQ4 What is the impact of curriculum design on final policy performance under safety constraints?
• RQ5 How can the teacher efficiently optimize curricula in online settings with limited supervision?

Key findings

• A curriculum of safety interventions can keep the student safe during learning by resetting to safe states when danger is detected.
• Under certain conditions, learning within intervention-induced CMDPs yields policies feasible in the original CMDP after the teacher is removed.
• A data-driven, online teacher improves curriculum policies across generations of students using observed progress statistics.
• Empirical results in Frozen Lake and Lunar Lander show curriculum-optimized CISR achieves comparable or superior final rewards with maintained safety compared to non-curriculum or fixed-intervention baselines.
• Curriculum policies learned by CISR transfer well across agents with different architectures and sensing capabilities.