[Paper Review] Reducing Commitment to Tasks with Off-Policy Hierarchical Reinforcement Learning

Mitchell Keith Bloch|arXiv (Cornell University)|Apr 27, 2011

Reinforcement Learning in Robotics7 references3 citations

TL;DR

This paper proposes off-policy hierarchical reinforcement learning methods that eliminate unintended on-policy updates by modifying temporal difference (TD) learning in HRL. It demonstrates that reducing commitment to subtask completion—through gated temporal second difference traces (GTSDT) and one-step intra-option learning—improves both online learning efficiency and final policy quality in the taxi-cab domain, challenging the long-held belief that subtask commitment is essential for efficient HRL learning.

ABSTRACT

In experimenting with off-policy temporal difference (TD) methods in hierarchical reinforcement learning (HRL) systems, we have observed unwanted on-policy learning under reproducible conditions. Here we present modifications to several TD methods that prevent unintentional on-policy learning from occurring. These modifications create a tension between exploration and learning. Traditional TD methods require commitment to finishing subtasks without exploration in order to update Q-values for early actions with high probability. One-step intra-option learning and temporal second difference traces (TSDT) do not suffer from this limitation. We demonstrate that our HRL system is efficient without commitment to completion of subtasks in a cliff-walking domain, contrary to a widespread claim in the literature that it is critical for efficiency of learning. Furthermore, decreasing commitment as exploration progresses is shown to improve both online performance and the resultant policy in the taxicab domain, opening a new avenue for research into when it is more beneficial to continue with the current subtask or to replan.

Motivation & Objective

• To address unintended on-policy learning in off-policy hierarchical reinforcement learning (HRL) systems.
• To challenge the widely held claim that commitment to completing subtasks is essential for efficient HRL learning.
• To develop TD methods that support reliable off-policy learning without requiring subtasks to converge first.
• To explore the impact of dynamically reducing commitment during exploration on learning performance.

Proposed method

• Modifies traditional TD methods to prevent on-policy updates during off-policy learning in HRL.
• Introduces gated temporal second difference traces (GTSDT), a corrected off-policy TD method for hierarchical options.
• Employs one-step intra-option learning (OSIO) to ensure consistent value backups during non-greedy subtask execution.
• Uses all-states updating to improve learning efficiency by updating Q-values for all goals in the hierarchy.
• Applies Boltzmann exploration with a cooling schedule to gradually reduce commitment from 1 to 0 over 100,000 episodes.
• Implements a hierarchical agent with shared Q-values based on task goals and non-hierarchical execution for online learning.

Experimental results

Research questions

• RQ1 Can off-policy HRL learn efficiently without requiring subtasks to be completed?
• RQ2 Does reducing commitment to subtask completion improve learning speed and final policy quality?
• RQ3 Are traditional TD methods like Q(0) and OSIO reliable for off-policy HRL when non-greedy actions are taken in subtasks?
• RQ4 Can GTSDT and OSIO support effective learning during non-hierarchical execution?
• RQ5 When is commitment to subtask completion more or less beneficial for learning efficiency?

Key findings

• GTSDT outperforms Fixed Q(0) and Fixed OSIO in terms of final policy quality, regardless of commitment level.
• Online performance is consistently best with GTSDT, and it improves significantly when commitment is reduced.
• Reducing commitment from 1 to 0 over 100,000 episodes improves the final policy for all three algorithms tested.
• Online performance improves for GTSDT and OSIO when commitment is reduced, while only slightly degrading for Fixed Q(0).
• The results contradict the literature claim that subtask completion commitment is critical for efficient HRL learning.
• The study demonstrates that reliable off-policy learning is possible without requiring subtasks to converge first, using corrected TD methods.