[Paper Review] Deep Inverse Q-learning with Constraints

Gabriel Kalweit, Maria Huegle+2 more|arXiv (Cornell University)|Aug 4, 2020

Reinforcement Learning in Robotics22 references4 citations

TL;DR

This paper introduces Deep Inverse Q-learning with Constraints, a novel IRL framework that bypasses the computationally expensive inner loop of Maximum Entropy IRL by solving the MDP only once via Inverse Action-value Iteration (IAVI), enabling closed-form reward recovery. It achieves up to several orders of magnitude speedup and supports nonlinear reward functions and hard constraints, demonstrating strong performance in autonomous lane-changing with only 30 minutes of demonstrations.

ABSTRACT

Popular Maximum Entropy Inverse Reinforcement Learning approaches require the computation of expected state visitation frequencies for the optimal policy under an estimate of the reward function. This usually requires intermediate value estimation in the inner loop of the algorithm, slowing down convergence considerably. In this work, we introduce a novel class of algorithms that only needs to solve the MDP underlying the demonstrated behavior once to recover the expert policy. This is possible through a formulation that exploits a probabilistic behavior assumption for the demonstrations within the structure of Q-learning. We propose Inverse Action-value Iteration which is able to fully recover an underlying reward of an external agent in closed-form analytically. We further provide an accompanying class of sampling-based variants which do not depend on a model of the environment. We show how to extend this class of algorithms to continuous state-spaces via function approximation and how to estimate a corresponding action-value function, leading to a policy as close as possible to the policy of the external agent, while optionally satisfying a list of predefined hard constraints. We evaluate the resulting algorithms called Inverse Action-value Iteration, Inverse Q-learning and Deep Inverse Q-learning on the Objectworld benchmark, showing a speedup of up to several orders of magnitude compared to (Deep) Max-Entropy algorithms. We further apply Deep Constrained Inverse Q-learning on the task of learning autonomous lane-changes in the open-source simulator SUMO achieving competent driving after training on data corresponding to 30 minutes of demonstrations.

Motivation & Objective

• To address the high computational cost of Maximum Entropy IRL, which requires repeated MDP solving in an inner loop.
• To enable closed-form recovery of the expert's reward function by modeling demonstrations as a Boltzmann policy over optimal Q-values.
• To develop model-free, sampling-based variants of IRL that support arbitrary nonlinear reward functions without inner-loop optimization.
• To extend the framework to continuous state spaces using function approximation for real-world applicability.
• To enforce hard constraints on behavior even when demonstrations violate them, enabling safe imitation in safety-critical domains.

Proposed method

• Proposes Inverse Action-value Iteration (IAVI), which formulates IRL as solving a system of linear equations derived from the MDP structure and Boltzmann policy assumption.
• Uses a probabilistic behavior assumption where actions are chosen according to a Boltzmann distribution over Q-values, enabling analytical recovery of the reward function.
• Introduces Inverse Q-learning (IQL) as a sampling-based, model-free variant using shifted Q-functions to avoid inner-loop MDP solving.
• Extends the framework to continuous states via deep function approximation, resulting in Deep Inverse Q-learning (DIQL).
• Introduces Constrained Inverse Q-learning (CIQL) and Deep Constrained Inverse Q-learning (DCIQL) to enforce predefined hard constraints during imitation.
• Employs stochastic approximation and function approximation to scale to high-dimensional and continuous environments.

Experimental results

Research questions

• RQ1 Can IRL be accelerated by eliminating the need for repeated MDP solving in the inner loop?
• RQ2 Can a closed-form solution for the expert reward be derived under a Boltzmann policy assumption over Q-values?
• RQ3 Can model-free IRL support arbitrary nonlinear reward functions without inner-loop optimization?
• RQ4 Can constraints be enforced on the policy even when demonstrations violate them?
• RQ5 Can the learned reward function enable faster and more data-efficient policy training than using the true immediate reward?

Key findings

• The proposed IAVI algorithm achieves up to several orders of magnitude speedup over MaxEnt IRL by solving the MDP only once.
• The learned reward function from IAVI enables faster policy convergence than training with the true immediate reward, suggesting a bootstrapping advantage.
• Deep Constrained Inverse Q-learning (DCIQL) successfully learns safe, constrained lane-changing behaviors from 30 minutes of unconstrained demonstrations in SUMO simulation.
• DCIQL achieves robust performance with only 1,000 expert demonstration samples (~1.5 hours of driving), demonstrating high data efficiency.
• The DCIQL agent satisfies both Keep Right and Safety constraints at all times, while still imitating the expert’s anticipatory overtaking behavior.
• Compared to CDQN, the agent trained on the IRL-learned reward converges significantly faster, highlighting the benefit of the inverse learning formulation.