[Paper Review] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey, Aravind Rajeswaran+3 more|arXiv (Cornell University)|Nov 5, 2018

Reinforcement Learning in RoboticsComputer Science36 references66 citations

TL;DR

POLO couples online trajectory optimization with offline value function learning and uncertainty-driven exploration to enable efficient, planning-based learning in high-dimensional control tasks.

ABSTRACT

We propose a plan online and learn offline (POLO) framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex simulated control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

Motivation & Objective

• Motivate continual acting and learning with an internal dynamics model in complex worlds.
• Show how local trajectory optimization interacts with global value function learning to stabilize and accelerate learning.
• Demonstrate that approximate value functions can reduce planning horizons and improve policy quality.
• Develop an exploration strategy that uses trajectory optimization to perform temporally coordinated exploration.

Proposed method

• Use model-based trajectory optimization (MPC) to compute locally optimal action sequences based on a nominal dynamics model.
• Apply fitted value iteration with parametric function approximators to learn the global value function V for guidance.
• Incorporate uncertainty-aware exploration by maintaining multiple value-function approximators and using a softmax over their outputs to form an optimistic value estimate.
• Plan to explore by optimizing trajectories under a posterior over value functions, enabling temporally coordinated exploration.
• Define N-step trajectory-based targets for value function updates to accelerate learning and stabilize training (Eq. 7).
• Iteratively collect experience, update an ensemble of value functions, and execute MPC with an optimistic terminal value.

Experimental results

Research questions

• RQ1 Does trajectory optimization combined with uncertainty estimation enable temporally coordinated exploration?
• RQ2 Can a learned value function allow MPC to use shorter planning horizons without sacrificing performance?
• RQ3 Does trajectory optimization accelerate and stabilize value-function learning in high-dimensional tasks?
• RQ4 Can POLO solve complex tasks (e.g., humanoid locomotion, dexterous manipulation) with limited real-world experience?

Key findings

• Trajectory optimization enables directed, temporally coordinated exploration that improves region coverage in state space.
• POLO-dominated planning horizons over plain MPC in high-dimensional tasks, showing faster skill acquisition and better performance.
• Longer horizon planning with MPC tolerates value-function approximation errors more robustly than greedy policies.
• N-step trajectory optimization accelerates value-function learning and stabilizes targets.
• A learned value function can guide MPC to achieve task progress even when rewards are sparse or changing.