[Paper Review] Iterative Hierarchical Optimization for Misspecified Problems (IHOMP)

Daniel J. Mankowitz, Timothy Mann+1 more|arXiv (Cornell University)|Feb 10, 2016

Reinforcement Learning in Robotics23 references11 citations

TL;DR

This paper proposes Iterative Hierarchical Optimization for Misspecified Problems (IHOMP), a method that iteratively learns context-specialized options and combines them via an inter-option policy to solve reinforcement learning problems where standard function approximation fails due to poor state representation. IHOMP provides theoretical convergence guarantees and extends to IHOMP-ROI, which jointly learns options and improves state-space partitions, enabling effective option reuse and significantly outperforming flat policies on misspecified tasks like Pinball and Two Rooms.

ABSTRACT

For complex, high-dimensional Markov Decision Processes (MDPs), it may be necessary to represent the policy with function approximation. A problem is misspecified whenever, the representation cannot express any policy with acceptable performance. We introduce IHOMP : an approach for solving misspecified problems. IHOMP iteratively learns a set of context specialized options and combines these options to solve an otherwise misspecified problem. Our main contribution is proving that IHOMP enjoys theoretical convergence guarantees. In addition, we extend IHOMP to exploit Option Interruption (OI) enabling it to decide where the learned options can be reused. Our experiments demonstrate that IHOMP can find near-optimal solutions to otherwise misspecified problems and that OI can further improve the solutions.

Motivation & Objective

• Address the challenge of solving reinforcement learning problems that are misspecified due to inadequate state representations, where standard function approximation fails to express high-performing policies.
• Develop a hierarchical reinforcement learning approach that mitigates misspecification by learning specialized options over partitioned state subregions, enabling better policy generalization.
• Provide theoretical convergence guarantees for the iterative learning of options in continuous state spaces under misspecification.
• Extend the framework to automatically improve state-space partitioning via Option Interruption, enabling discovery of optimal option reuse regions.
• Demonstrate that the method can find near-optimal solutions in complex, high-dimensional domains where flat policy representations fail.

Proposed method

• IHOMP uses a partitioned state space, training one option per partition class using a black-box RL algorithm, with options initialized arbitrarily and updated iteratively.
• Each option is trained to maximize cumulative reward within its partition, and reward signals propagate backward across partitions through the inter-option policy.
• The inter-option policy combines the outputs of specialized options to generate actions, enabling coordinated behavior across state subregions.
• The algorithm iteratively refines option policies, allowing options in low-reward regions to benefit from reward propagation from better-performing options.
• IHOMP-ROI integrates regularized option interruption to jointly learn optimal option policies and state-space partitions, discovering where options can be reused.
• The method supports arbitrary partitioning schemes and is compatible with linear and non-linear function approximators, including deep Q-networks.

Experimental results

Research questions

• RQ1 Can iterative learning of context-specialized options in a hierarchical framework solve reinforcement learning problems that are misspecified due to limited state representations?
• RQ2 Does the proposed IHOMP algorithm provide theoretical convergence guarantees for solving misspecified MDPs in continuous state spaces?
• RQ3 Can Option Interruption be effectively used to improve state-space partitioning and enable intelligent option reuse in hierarchical RL?
• RQ4 How does IHOMP perform compared to flat policy approaches in real-world misspecified environments like Pinball and Two Rooms?
• RQ5 To what extent can IHOMP-ROI discover optimal partitioning and reuse regions without prior knowledge of the state space structure?

Key findings

• IHOMP successfully finds near-optimal solutions in misspecified problems such as the S-shaped domain and the Two Rooms environment, where flat policies fail entirely.
• In the Pinball world task, IHOMP with a 4×3×1×1 partition outperformed the flat policy, achieving a stable average reward after a single iteration due to goal-proximal option initialization.
• IHOMP-ROI learned a non-trivial, optimal partitioning in the Two Rooms domain, enabling the agent to navigate between rooms despite a limited policy representation.
• The learned partition in IHOMP-ROI included counterintuitive option reuse—executing the red option in region B—demonstrating the algorithm's ability to discover effective reuse patterns.
• The performance gap between IHOMP and flat policies was significant, with IHOMP-ROI achieving higher average rewards and solving tasks that were otherwise intractable.
• The cost of partitioning was sensitive to design, with coarser partitions (e.g., 3×3) outperforming finer ones (e.g., 4×4) in Puddle World, highlighting the importance of partition quality.