[Paper Review] Model Primitive Hierarchical Lifelong Reinforcement Learning

Bo-Han Wu, Jayesh K. Gupta+1 more|arXiv (Cornell University)|Mar 4, 2019

Reinforcement Learning in Robotics26 references4 citations

TL;DR

This paper proposes Model Primitive Hierarchical Lifelong Reinforcement Learning (MPHRL), a framework that uses diverse suboptimal world models—called model primitives—to automatically decompose complex tasks into modular subpolicies and a gating controller in a bottom-up manner. The approach enables efficient lifelong learning in high-dimensional continuous control tasks by transferring subpolicies and avoiding catastrophic forgetting, achieving significant sample efficiency gains over standard PPO.

ABSTRACT

Learning interpretable and transferable subpolicies and performing task decomposition from a single, complex task is difficult. Some traditional hierarchical reinforcement learning techniques enforce this decomposition in a top-down manner, while meta-learning techniques require a task distribution at hand to learn such decompositions. This paper presents a framework for using diverse suboptimal world models to decompose complex task solutions into simpler modular subpolicies. This framework performs automatic decomposition of a single source task in a bottom up manner, concurrently learning the required modular subpolicies as well as a controller to coordinate them. We perform a series of experiments on high dimensional continuous action control tasks to demonstrate the effectiveness of this approach at both complex single task learning and lifelong learning. Finally, we perform ablation studies to understand the importance and robustness of different elements in the framework and limitations to this approach.

Motivation & Objective

• To address the challenge of learning interpretable, transferable subpolicies for complex tasks without requiring a predefined task distribution or accurate world models.
• To enable lifelong reinforcement learning by decomposing a single complex task into modular subpolicies that can be reused across related tasks.
• To reduce sample complexity and prevent catastrophic forgetting in sequential task learning by leveraging model primitives as a basis for hierarchical policy decomposition.
• To demonstrate that suboptimal, diverse world models can serve as effective primitives for task decomposition and transfer learning.

Proposed method

• The framework uses a set of suboptimal world models—termed model primitives—each specialized to a region of the state space, to guide the decomposition of a complex task into simpler subtasks.
• Each model primitive predicts the next state given current state and action, and is trained using a weighted mean squared error loss on dynamics prediction.
• A gating controller is trained jointly with subpolicies to select, sequence, and adapt subpolicies based on current state and model primitive predictions.
• The method employs a mixture-of-experts-like architecture where the gating controller uses a cross-entropy target derived from predicted model primitive probabilities.
• Subpolicies are trained end-to-end with the gating controller using policy gradient methods, with the controller learning to route actions through appropriate subpolicies.
• The framework avoids reliance on accurate world models or predefined task distributions, instead leveraging the diversity of model primitives to enable robust decomposition.

Experimental results

Research questions

• RQ1 Can suboptimal, diverse world models be effectively used to decompose a single complex task into reusable, modular subpolicies?
• RQ2 How does the joint training of subpolicies and a gating controller improve sample efficiency in lifelong reinforcement learning?
• RQ3 To what extent does the use of model primitives mitigate catastrophic forgetting and negative transfer in sequential task learning?
• RQ4 How robust is the framework to imperfect or noisy model primitives, and does it maintain performance when model primitives are learned rather than hand-designed?

Key findings

• MPHRL achieved substantial sample efficiency improvements over standard PPO, reducing the number of timesteps required to reach 80% success rate by more than 50% on the 10-Maze environment.
• When subpolicies were re-initialized for each new task, sample complexity more than quintupled, demonstrating the critical role of subpolicy transfer in efficiency gains.
• The gating controller without re-initialization during transfer showed increased variance in performance, indicating volatility due to negative transfer, highlighting the importance of proper controller management.
• Using learned model primitives instead of hand-designed ones resulted in performance slightly below hand-designed models but still significantly outperforming standard PPO.
• Failure occurred when using the incorrect cross-entropy target in Eq. 1 instead of Eq. 9, with all runs failing to solve the first five tasks, indicating the necessity of coupling the cross-entropy target with the predicted model probabilities.
• The framework demonstrated robustness to task ordering and partial decomposition, maintaining high performance even when the source task did not allow full decomposition into all useful subpolicies.