[Paper Review] Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

Yevgen Chebotar, Karol Hausman+4 more|arXiv (Cornell University)|Mar 8, 2017

Reinforcement Learning in Robotics27 references86 citations

TL;DR

PILQR integrates model-based LQR-FLM updates with model-free PI 2 updates for trajectory-centric policies, enabling data-efficient learning and effective training of neural networks via GPS in both simulations and real robots.

ABSTRACT

Reinforcement learning (RL) algorithms for real-world robotic applications need a data-efficient learning process and the ability to handle complex, unknown dynamical systems. These requirements are handled well by model-based and model-free RL approaches, respectively. In this work, we aim to combine the advantages of these two types of methods in a principled manner. By focusing on time-varying linear-Gaussian policies, we enable a model-based algorithm based on the linear quadratic regulator (LQR) that can be integrated into the model-free framework of path integral policy improvement (PI2). We can further combine our method with guided policy search (GPS) to train arbitrary parameterized policies such as deep neural networks. Our simulation and real-world experiments demonstrate that this method can solve challenging manipulation tasks with comparable or better performance than model-free methods while maintaining the sample efficiency of model-based methods. A video presenting our results is available at https://sites.google.com/site/icml17pilqr

Motivation & Objective

• Motivate data-efficient reinforcement learning for real-world robots with unknown dynamics.
• Leverage time-varying linear-Gaussian policies to enable fast model-based updates.
• Combine model-based updates with model-free corrections within a single trajectory-centric framework.
• Integrate PILQR into guided policy search to train general-purpose neural network policies.
• Demonstrate effectiveness on simulation tasks and real-world robotic manipulation tasks.

Proposed method

• Use time-varying linear-Gaussian (TVLG) policies p(u_t|x_t) with dynamics p(x_{t+1}|x_t,u_t) modeled as Gaussian.
• Extend LQR with fitted linear models (LQR-FLM) to perform KL-constrained, second-order updates using a local quadratic cost approximation.
• Adopt Path Integral policy improvement (PI 2) as a model-free, KL-constrained update driven by cost-to-go S and a soft-max reweighting of trajectories.
• Introduce a two-stage PI 2 update: first update with the model-based cost approximation, then update with the residual cost using PI 2.
• Decompose the PI 2 update into a model-based approximation hat{S} and a residual tilde{S}, and perform updates sequentially as in equations (5)–(6).
• Incorporate a two-stage PILQR algorithm (Algorithm 1): generate trajectories, fit TVLG dynamics, compute hat{c} and tilde{c}, adjust KL step epsilon_t, compute eta_t, perform LQR-FLM on hat{c}, then PI 2 on tilde{c}.

Experimental results

Research questions

• RQ1 Does PILQR achieve better final performance and sample efficiency than model-based or model-free baselines on trajectory-centric tasks?
• RQ2 Can PILQR train high-dimensional neural network policies via GPS with sample efficiency comparable to model-based methods?
• RQ3 Is PILQR capable of solving complex manipulation tasks with discontinuous dynamics in both simulation and on real robotic platforms?
• RQ4 How does combining LQR-FLM updates with PI 2 corrections affect learning stability and convergence?

Key findings

• PILQR outperforms LQR-FLM and PI 2 on challenging gripper pusher and door opening tasks in simulation, solving all conditions with about 400 episodes per condition.
• MDGPS with PILQR trains neural network policies that reach comparable performance to deep RL baselines while using orders of magnitude fewer samples on certain tasks.
• In real robot hockey and power plug plugging tasks, PILQR learns robust policies from scratch without demonstrations within less than an hour of experience.
• PILQR enables neural network policies to generalize across multiple goal locations in hockey with 90% success across varied targets.
• PILQR demonstrates most favorable success where PI 2 alone struggles, and LQR-FLM alone fails on tasks with discontinuous dynamics, highlighting the benefit of the hybrid approach.