[Paper Review] Learning Shared Safety Constraints from Multi-task Demonstrations

Konwoo Kim, Gokul Swamy+4 more|arXiv (Cornell University)|Sep 1, 2023

Reinforcement Learning in RoboticsComputer Science3 citations

TL;DR

This paper proposes multi-task inverse constraint learning (MT-ICL), a method that leverages diverse expert demonstrations across multiple tasks to learn shared safety constraints more robustly than single-task approaches. By framing inverse constraint learning as a zero-sum game and using multi-task data to avoid overly conservative constraints, the method successfully recovers ground-truth safety constraints in continuous control environments, matching expert performance without environment interaction.

ABSTRACT

Regardless of the particular task we want them to perform in an environment, there are often shared safety constraints we want our agents to respect. For example, regardless of whether it is making a sandwich or clearing the table, a kitchen robot should not break a plate. Manually specifying such a constraint can be both time-consuming and error-prone. We show how to learn constraints from expert demonstrations of safe task completion by extending inverse reinforcement learning (IRL) techniques to the space of constraints. Intuitively, we learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to. Unfortunately, the constraint learning problem is rather ill-posed and typically leads to overly conservative constraints that forbid all behavior that the expert did not take. We counter this by leveraging diverse demonstrations that naturally occur in multi-task settings to learn a tighter set of constraints. We validate our method with simulation experiments on high-dimensional continuous control tasks.

Motivation & Objective

• To address the challenge of learning implicit, task-agnostic safety constraints that are difficult to manually specify.
• To overcome the ill-posed nature of inverse constraint learning, which often leads to overly conservative constraints.
• To improve constraint generalization and robustness by leveraging diverse demonstrations from multiple tasks.
• To formalize a multi-task extension of inverse constraint learning that enables safe and high-performing policies across unseen tasks.
• To validate the method on high-dimensional continuous control tasks, demonstrating recovery of ground-truth constraints without environment interaction.

Proposed method

• Formalize inverse constraint learning (ICL) as a two-player zero-sum game between a policy player (maximizing task reward under a constraint) and a constraint player (selecting constraints that penalize the policy relative to the expert).
• Extend ICL to a multi-task setting by introducing a constraint player that evaluates constraints across multiple task-specific policy players, using aggregate expert and learner data to avoid degenerate solutions.
• Use a constrained reinforcement learning (CRL) oracle in the inner loop to find the optimal policy under a candidate constraint, enabling safe policy optimization.
• Leverage multi-task demonstration data to improve coverage of the state space, reducing overfitting to idiosyncratic expert behaviors in single tasks.
• Train a deep neural network to represent the constraint function, enabling generalization across complex, high-dimensional environments.
• Iteratively refine the constraint using a game-theoretic optimization process, converging to a constraint that forbids only high-reward, unsafe behaviors the expert avoided.

Figure 1 : A visual depiction of the optimization problem we’re trying to solve in ICL . We attempt to pick a constraint that minimizes the value difference over the expert policy a safe policy could have. The star corresponds to the output of CRL.

Experimental results

Research questions

• RQ1 Can multi-task demonstrations improve the robustness and generalization of learned safety constraints compared to single-task ICL?
• RQ2 How does the proposed multi-task ICL framework avoid the overly conservative constraints common in prior ICL methods?
• RQ3 To what extent can the method recover ground-truth safety constraints in continuous control environments without interacting with the true environment?
• RQ4 Can the method generalize to unseen tasks by learning a shared constraint from diverse task demonstrations?
• RQ5 What is the impact of expert sub-optimality on constraint learning, and can the method remain robust under noisy demonstrations?

Key findings

• The multi-task ICL method successfully recovers the ground-truth maze structure in the AntMaze environment using only expert demonstrations, without any environment interaction.
• In the AntMaze task, the multi-task constraint network (g)/(h) approximately recovers the true walls, while single-task networks (d) and ensemble methods (e)/(f) learn spurious, incorrect constraints.
• The method achieves expert-level performance and constraint violation rates across all 10 tasks in the AntMaze environment, with results averaged across 3 seeds and standard errors computed.
• Even with noisy expert demonstrations (Gaussian noise on actions), the method maintains safe and performant policies, demonstrating robustness to sub-optimal experts.
• The approach generalizes to unseen tasks by learning a shared constraint from multi-task data, as shown by consistent performance across diverse navigation goals.
• The zero-sum game formulation enables stable constraint learning, with convergence observed over iterations in both simulated and continuous control settings.

Figure 2 : If we have a sufficient diversity of expert policies, none of which are optimal along the reward vector, we can identify the hyperplane that separates the safe policies from the unsafe policies. The constraint (red, dashed) will be orthogonal to this hyperplane. For this example, because