[Paper Review] Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

Thomas G. Dietterich|ArXiv.org|May 21, 1999

Reinforcement Learning in Robotics27 references21 citations

TL;DR

This paper introduces MAXQ, a hierarchical reinforcement learning framework that decomposes the value function of a Markov decision process into additive components across a hierarchy of subtasks. It enables efficient learning via state abstraction and subtask reuse, and proves convergence to a recursively optimal policy, with experiments showing faster convergence and improved performance over flat Q-learning.

ABSTRACT

This paper presents the MAXQ approach to hierarchical reinforcement learning based on decomposing the target Markov decision process (MDP) into a hierarchy of smaller MDPs and decomposing the value function of the target MDP into an additive combination of the value functions of the smaller MDPs. The paper defines the MAXQ hierarchy, proves formal results on its representational power, and establishes five conditions for the safe use of state abstractions. The paper presents an online model-free learning algorithm, MAXQ-Q, and proves that it converges wih probability 1 to a kind of locally-optimal policy known as a recursively optimal policy, even in the presence of the five kinds of state abstraction. The paper evaluates the MAXQ representation and MAXQ-Q through a series of experiments in three domains and shows experimentally that MAXQ-Q (with state abstractions) converges to a recursively optimal policy much faster than flat Q learning. The fact that MAXQ learns a representation of the value function has an important benefit: it makes it possible to compute and execute an improved, non-hierarchical policy via a procedure similar to the policy improvement step of policy iteration. The paper demonstrates the effectiveness of this non-hierarchical execution experimentally. Finally, the paper concludes with a comparison to related work and a discussion of the design tradeoffs in hierarchical reinforcement learning.

Motivation & Objective

• To address the scalability limitations of flat reinforcement learning in large state spaces by introducing a hierarchical value function decomposition.
• To formalize a hierarchical reinforcement learning method that supports subtask reuse, state abstraction, and efficient learning through procedural and declarative semantics.
• To develop a learning algorithm (MAXQ-Q) that converges to a recursively optimal policy even under state abstractions.
• To enable the derivation of improved non-hierarchical policies from the hierarchical value function representation.
• To establish formal conditions under which state abstractions can be safely applied in hierarchical RL without compromising convergence.

Proposed method

• MAXQ decomposes the value function of a target MDP into an additive combination of value functions for subtasks in a hierarchical graph structure.
• The hierarchy is defined via a directed acyclic graph (DAG) where internal nodes represent subgoals and leaves represent primitive actions.
• MAXQ-Q is an online, model-free learning algorithm that updates value functions at each node using Q-learning principles, propagating updates bottom-up.
• The method supports state abstraction through five formal conditions: Max Node Irrelevance, Leaf Irrelevance, Result Distribution Irrelevance, Shielding, and Termination.
• A policy improvement procedure is applied to the hierarchical value function to extract a non-hierarchical, improved policy.
• The framework allows subtask sharing and re-use by maintaining independent value functions for each subtask.

Experimental results

Research questions

• RQ1 Can a hierarchical value function decomposition represent the value function of any policy consistent with the given hierarchy?
• RQ2 Under what conditions can state abstractions be safely applied in hierarchical reinforcement learning without affecting convergence?
• RQ3 Does the MAXQ-Q learning algorithm converge to a locally optimal policy in the presence of state abstractions?
• RQ4 Can the hierarchical value function representation be used to derive a better, non-hierarchical policy than the original hierarchical one?
• RQ5 How does the performance of MAXQ-Q compare to flat Q-learning in terms of convergence speed and final policy quality?

Key findings

• MAXQ-Q converges with probability 1 to a recursively optimal policy, even when using state abstractions that ignore large parts of the state space.
• The MAXQ value function decomposition can represent the value function of any hierarchical policy under both finite-horizon undiscounted and infinite-horizon discounted reward criteria.
• Experiments in the Taxi and Kaelbling HDG tasks show that MAXQ-Q with state abstractions converges significantly faster than flat Q-learning.
• In the Fickle Taxi Task, non-hierarchical execution based on the MAXQ value function achieves optimal performance, demonstrating the effectiveness of policy improvement from the hierarchical representation.
• The HDG task experiment shows a substantial performance improvement when using the improved non-hierarchical policy derived from MAXQ, compared to the original hierarchical policy.
• The paper establishes five formal conditions (Max Node Irrelevance, Leaf Irrelevance, Result Distribution Irrelevance, Shielding, and Termination) that allow safe use of state abstractions in hierarchical RL.