[Paper Review] Exploring Model-based Planning with Policy Networks

Tingwu Wang, Jimmy Ba|arXiv (Cornell University)|Jun 20, 2019

Reinforcement Learning in Robotics39 references76 citations

TL;DR

POPLIN introduces model-based policy planning that optimizes either action sequences or policy parameters using neural networks, achieving state-of-the-art sample efficiency on MuJoCo tasks by planning in parameter space with a policy network initialization.

ABSTRACT

Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in terms of both sample efficiency and asymptotic performance. Despite their initial successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that POPLIN obtains state-of-the-art performance in the MuJoCo benchmarking environments, being about 3x more sample efficient than the state-of-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released in https://github.com/WilsonWangTHU/POPLIN.

Motivation & Objective

• Motivate improving sample efficiency in model-based RL for high-dimensional locomotion tasks.
• Propose a planning framework that uses policy networks to generate good proposals for online planning.
• Show that planning in policy parameter space yields smoother optimization surfaces and better search efficiency.
• Demonstrate state-of-the-art performance on MuJoCo benchmarks with significant sample efficiency gains.

Proposed method

• Define planning at each time-step as an optimization over either action sequences or policy parameters.
• POPLIN-A: use a policy network to propose action sequences and refine with cross-entropy method (CEM) in action space.
• POPLIN-P: perform planning in the policy parameter space by perturbing network parameters and evaluating resultant policies.
• Two distillation pathways: policy distillation (BC, GAN) and AVG-based updates to accumulate planning experience.
• Compare MPC-ready control (plan and execute first action) against direct policy control (execute policy output).
• Provide empirical analysis on the smoothness of optimization surfaces and the benefits of parameter-space planning.

Experimental results

Research questions

• RQ1 Can integrating a policy network with online planning improve sample efficiency over traditional random-shooting MPC methods like PETS?
• RQ2 Is planning in the policy parameter space (as opposed to action-space noise) easier due to a smoother optimization landscape?
• RQ3 How do different policy distillation strategies affect end-task performance and real-time control viability?

Key findings

• POPLIN achieves state-of-the-art performance on MuJoCo benchmarks, with approximately 3x better sample efficiency than PETS, TD3, and SAC.
• Planning in parameter space (POPLIN-P) yields a smoother optimization surface than planning in action space, enabling more effective search.
• Distilled policy networks can perform well at test time without expansive online planning in some environments (e.g., Cheetah).
• POPLIN-A excels in simpler tasks (Pendulum, Cart-pole, Swimmer) but is less advantageous than POPLIN-P on more complex tasks (Ant, Cheetah, Hopper).
• POPLIN-P variants (Uni, Sep, Avg, GAN, BC) show varying strengths across environments, with POPLIN-P-Sep often outperforming POPLIN-P-Uni in planning efficiency.