[Paper Review] FeUdal Networks for Hierarchical Reinforcement Learning

Alexander Sasha Vezhnevets, Simon Osindero+5 more|arXiv (Cornell University)|Mar 3, 2017

Reinforcement Learning in RoboticsComputer Science252 citations

TL;DR

FeUdal Networks (FuN) introduce a two-level hierarchy with a Manager setting directional, latent goals at a slow timescale and a Worker executing primitive actions at every tick, trained via a transition-policy gradient and intrinsic rewards to improve long-term credit assignment and memory.

ABSTRACT

We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels -- allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits -- in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.

Motivation & Objective

• Motivate and address long-term credit assignment and memory challenges in reinforcement learning.
• Propose a hierarchical, differentiable architecture that decouples learning across levels and time scales.
• Introduce a novel transition policy gradient update that gives semantic meaning to Manager goals.
• Encourage emergence of reusable sub-policies through directional goals and intrinsic motivation for the Worker.
• Show empirical benefits on ATARI and 3D DeepMind Lab tasks against strong baselines.

Proposed method

• Two-level FuN architecture with a Manager that sets goals and a Worker that acts conditioned on those goals.
• Manager operates at a lower temporal resolution and uses a dilated LSTM to maintain long-horizon memory.
• Worker receives intrinsic reward to follow Manager goals and uses an advantage actor-critic update.
• Goals are embedded in a low-dimensional space and modulate Worker policies via a multiplicative interaction.
• Manager learning follows an approximate transition policy gradient based on cosine similarity of latent state transitions.
• Intrinsic reward for the Worker is a running average cosine similarity encouraging progress toward goal directions.
• Transition model assumption uses a von Mises-Fisher directional distribution to justify the Manager’s gradient update.

Experimental results

Research questions

• RQ1 Can a hierarchical, differentiable FuN architecture improve long-term credit assignment and memory in reinforcement learning tasks?
• RQ2 Do directional, latent goals and decoupled learning yield interpretable sub-policies that transfer across time scales?
• RQ3 Does a transition-policy gradient provide meaningful training signals for the Manager to optimize extrinsic reward?
• RQ4 Is intrinsic motivation essential for the Worker to effectively realize Manager goals?
• RQ5 How does FuN compare to strong LSTM baselines and to end-to-end options-based approaches on Atari and DeepMind Lab tasks?

Key findings

• FuN significantly improves long-term credit assignment and memorisation on Atari games including Montezuma’s Revenge and in DeepMind Lab tasks.
• The Manager learns meaningful, directional sub-goals that guide the Worker via intrinsic rewards.
• FuN outperforms an LSTM baseline on multiple Atari games and Memory tasks, especially when long BPTT unrolls are used.
• A dilated LSTM (dLSTM) in the Manager enables gradients over hundreds of time steps and supports long-horizon planning.
• Ablation studies show the transition policy gradient, directional goals, and intrinsic motivation are crucial for performance.
• FuN’s Manager-Worker separation allows potential transfer of the transition policy across embodiments and action repeats.