[Paper Review] Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Chelsea Finn, Sergey Levine+1 more|arXiv (Cornell University)|Mar 1, 2016

Optical Imaging and Spectroscopy Techniques369 citations

TL;DR

This paper introduces Guided Cost Learning, which learns nonlinear cost functions (e.g., neural networks) from demonstrations by integrating maximum entropy IOC with policy optimization to handle unknown dynamics and high-dimensional systems, achieving improved task performance and sample efficiency on robotic tasks.

ABSTRACT

Reinforcement learning can acquire complex behaviors from high-level specifications. However, defining a cost function that can be optimized effectively and encodes the correct task is challenging in practice. We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems. Our method addresses two key challenges in inverse optimal control: first, the need for informative features and effective regularization to impose structure on the cost, and second, the difficulty of learning the cost function under unknown dynamics for high-dimensional continuous systems. To address the former challenge, we present an algorithm capable of learning arbitrary nonlinear cost functions, such as neural networks, without meticulous feature engineering. To address the latter challenge, we formulate an efficient sample-based approximation for MaxEnt IOC. We evaluate our method on a series of simulated tasks and real-world robotic manipulation problems, demonstrating substantial improvement over prior methods both in terms of task complexity and sample efficiency.

Motivation & Objective

• Motivate learning task costs from expert demonstrations to enable complex robotic behaviors without hand-designed features.
• Overcome underdefinition in IOC by enabling expressive nonlinear cost representations (neural networks).
• Address unknown dynamics and high-dimensional state/action spaces via sample-based IOC and policy optimization.
• Reduce engineering burden by jointly learning cost and controller, enabling practical deployment on real robots.

Proposed method

• Adopt maximum entropy IOC where demonstrations are near-optimal under an unknown cost c_theta.
• Represent c_theta with neural networks operating on raw state inputs to avoid hand-crafted features.
• Use a sample-based approximation of the IOC objective with importance sampling to estimate the partition function Z.
• Interleave IOC optimization with a policy optimization procedure that adapts the trajectory distribution q(tau) toward exp(-c_theta(tau)).
• Update the cost parameters theta via stochastic gradients using demonstrations and trajectory samples.
• Regularize the nonlinear cost with locally constant rate (lcr) and monotonicity (mono) terms to mitigate overfitting.
• Provide a learned time-varying linear-Gaussian controller q(u_t|x_t) that implements the learned policy.

Experimental results

Research questions

• RQ1 Can expressive nonlinear cost functions learn from demonstrations in IOC settings with unknown dynamics?
• RQ2 Does adaptive sampling via policy optimization improve the estimation of the IOC partition function and final cost quality?
• RQ3 Can the method handle high-dimensional robotic systems and real-world torque-controlled tasks without hand-crafted features?
• RQ4 Do regularization techniques improve generalization and prevent overfitting of learned costs in episodic robotic tasks?
• RQ5 Is it beneficial to learn both the cost and the controller for new task instances rather than reuse a fixed cost?

Key findings

• The method learns nonlinear costs (e.g., neural networks) from demonstrations and outperforms prior IOC methods on simulated tasks with unknown dynamics.
• Guided cost learning coupled with policy optimization improves sample efficiency and supports high-dimensional robotic tasks (e.g., 7-DOF arm peg insertion).
• Regularization terms (local constant rate and monotonicity) help mitigate overfitting and improve learning in episodic tasks.
• On real PR2 robot tasks, guided cost learning achieved substantial success where affine costs failed, including dish placement and pouring with visual features.
• The approach yields both a learned cost function and a corresponding controller suitable for executing learned behaviors on new task instances.