[Paper Review] Learning Soft Constraints From Constrained Expert Demonstrations

Gaurav Ashish, Kasra Rezaee+2 more|arXiv (Cornell University)|Jun 2, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a novel inverse constraint learning (ICL) method that recovers cumulative soft constraints—constraints satisfied in expectation rather than strictly at every step—from expert demonstrations, assuming a known reward function. Using a constrained Markov decision process (CMDP) framework and iterative optimization, the approach learns flexible neural network-based constraint functions that generalize across synthetic, robotic, and real-world highway driving scenarios with high accuracy and robustness to noise.

ABSTRACT

Inverse reinforcement learning (IRL) methods assume that the expert data is generated by an agent optimizing some reward function. However, in many settings, the agent may optimize a reward function subject to some constraints, where the constraints induce behaviors that may be otherwise difficult to express with just a reward function. We consider the setting where the reward function is given, and the constraints are unknown, and propose a method that is able to recover these constraints satisfactorily from the expert data. While previous work has focused on recovering hard constraints, our method can recover cumulative soft constraints that the agent satisfies on average per episode. In IRL fashion, our method solves this problem by adjusting the constraint function iteratively through a constrained optimization procedure, until the agent behavior matches the expert behavior. We demonstrate our approach on synthetic environments, robotics environments and real world highway driving scenarios.

Motivation & Objective

• To address the limitation of existing inverse reinforcement learning (IRL) methods that cannot recover constraints, especially in safety-critical applications where constraints are essential.
• To learn cumulative soft constraints—violations allowed in individual trajectories but bounded in expectation—rather than hard constraints that must be satisfied in every trajectory.
• To develop a method that works with arbitrary state-action spaces, including continuous spaces, and learns flexible, neural network-based constraint functions.
• To demonstrate the method's effectiveness across diverse environments, including synthetic tasks, robotic control, and real-world highway driving scenarios.
• To provide a more interpretable and robust alternative to reward-only modeling by learning constraints that reflect safety and operational limits.

Proposed method

• The method formulates inverse constraint learning within the constrained MDP (CMDP) framework, where the agent maximizes expected reward subject to constraints on expected cumulative constraint values.
• It uses an iterative constrained optimization procedure to adjust the neural network-based constraint function until the agent's behavior matches the expert demonstrations.
• The constraint function is parameterized as a deep neural network and trained via a differentiable optimization process that minimizes the difference between expert and agent constraint accruals.
• The approach employs a dual optimization scheme similar to IRL, where the constraint function is updated based on the discrepancy between expert and agent constraint values over trajectories.
• The method is robust to noise and minor violations in expert data, as it only enforces constraints in expectation, not per-trajectory.
• It supports continuous state-action spaces and generalizes across diverse environments, including those with complex dynamics like robotic locomotion and highway driving.

Experimental results

Research questions

• RQ1 Can inverse constraint learning recover soft constraints that are satisfied in expectation rather than strictly at every step?
• RQ2 How well can a neural network-based constraint function generalize across diverse environments, including continuous control and real-world driving scenarios?
• RQ3 Does the proposed method outperform existing ICL approaches in terms of constraint accuracy and behavioral imitation?
• RQ4 How robust is the method to noise and minor violations in expert demonstrations?
• RQ5 Can the learned constraints be interpreted meaningfully and used to improve safety in real-world applications?

Key findings

• The method successfully recovers soft constraints in synthetic environments such as Gridworld and CartPole, achieving near-zero constraint mean squared error (CMSE) across multiple configurations.
• In the Ant and Half-Cheetah environments, the method achieved CMSE values of 0.01 ± 0.00 and 0.01 ± 0.00 respectively, indicating high-fidelity constraint recovery.
• For the HighD and ExiD highway driving scenarios, the method learned meaningful constraints related to speed and gap control, with normalized accruals closely matching expert behavior.
• The method demonstrated robustness to noise and minor violations in expert data, as it only enforces constraints in expectation, not per-trajectory.
• Training plots show consistent convergence of the constraint function and accrual values across 5 random seeds, indicating stable learning.
• The method outperformed baseline approaches like GAIL-Constraint and ICRL in constraint function accuracy and behavioral imitation, especially in complex environments.