[Paper Review] Learning Safety Constraints from Demonstrations with Unknown Rewards

David Lindner, Xin Chen+3 more|arXiv (Cornell University)|May 25, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces CoCoRL, a novel method for learning safety constraints in Constrained Markov Decision Processes (CMDPs) from demonstrations with unknown rewards and no knowledge of environment dynamics. By constructing a convex safe set from diverse, potentially sub-optimal demonstrations, CoCoRL guarantees safety and achieves asymptotic optimality for near-optimal policies, outperforming IRL-based baselines that fail to ensure safety.

ABSTRACT

We propose Convex Constraint Learning for Reinforcement Learning (CoCoRL), a novel approach for inferring shared constraints in a Constrained Markov Decision Process (CMDP) from a set of safe demonstrations with possibly different reward functions. While previous work is limited to demonstrations with known rewards or fully known environment dynamics, CoCoRL can learn constraints from demonstrations with different unknown rewards without knowledge of the environment dynamics. CoCoRL constructs a convex safe set based on demonstrations, which provably guarantees safety even for potentially sub-optimal (but safe) demonstrations. For near-optimal demonstrations, CoCoRL converges to the true safe set with no policy regret. We evaluate CoCoRL in gridworld environments and a driving simulation with multiple constraints. CoCoRL learns constraints that lead to safe driving behavior. Importantly, we can safely transfer the learned constraints to different tasks and environments. In contrast, alternative methods based on Inverse Reinforcement Learning (IRL) often exhibit poor performance and learn unsafe policies.

Motivation & Objective

• To address the challenge of learning safety constraints in CMDPs when reward functions are unknown and environment dynamics are unobserved.
• To develop a method that guarantees safety even for sub-optimal but safe demonstrations.
• To achieve asymptotic optimality for near-optimal demonstrations without relying on known rewards or dynamics.
• To enable transfer of learned constraints across different tasks and environments.
• To demonstrate that IRL-based approaches cannot guarantee safety in such settings.

Proposed method

• CoCoRL constructs a convex safe set in the feature space using demonstrations, ensuring all policies within this set satisfy the shared constraints.
• It formulates constraint inference as a convex optimization problem over the occupancy measures of demonstrated trajectories.
• The method leverages the fact that safe policies must lie within the intersection of half-spaces defined by the constraints, forming a convex polytope.
• For near-optimal demonstrations, CoCoRL converges to the true safe set with no policy regret, as proven via theoretical analysis.
• It does not require knowledge of the environment's transition dynamics or the reward functions of the demonstrators.
• The approach is evaluated in tabular environments and a continuous driving simulation with multiple constraints.

Experimental results

Research questions

• RQ1 Can safety constraints be reliably inferred from demonstrations when the underlying reward functions are unknown?
• RQ2 Can a constraint learning method guarantee safety even when demonstrations are sub-optimal?
• RQ3 Does the method achieve asymptotic optimality for near-optimal demonstrations without access to dynamics or rewards?
• RQ4 How does CoCoRL compare to IRL-based baselines in terms of safety and performance?
• RQ5 Can the inferred constraints be successfully transferred to new tasks and environments?

Key findings

• CoCoRL guarantees safety for all policies derived from the inferred convex safe set, with no unsafe solutions observed in experiments.
• In single-state CMDPs, CoCoRL achieves high return with a small number of demonstrations and converges to the optimal policy as the number of demonstrations increases.
• The sample complexity of CoCoRL scales with feature dimensionality, as predicted by Theorem 3, but is relatively insensitive to the number of true constraints.
• All IRL-based baselines—including maximum entropy and maximum margin variants—fail to ensure safety and return unsafe policies in both gridworld and driving environments.
• In the driving simulation, CoCoRL successfully learns constraints that lead to safe driving behavior and generalizes to new tasks and environments, while IRL baselines violate constraints and underperform.
• The Known Reward IRL baseline, which has access to true rewards, still fails to ensure safety, demonstrating that IRL cannot be relied upon for safety-critical applications.