[Paper Review] Language as an Abstraction for Hierarchical Deep Reinforcement Learning

Yiding Jiang, Shixiang Gu+2 more|arXiv (Cornell University)|Jun 18, 2019

Reinforcement Learning in Robotics65 references65 citations

TL;DR

The paper proposes HAL, a framework that uses language as the abstraction between high- and low-level policies in hierarchical RL, enabling compositional, instruction-following low-level policies and a high-level policy that operates in the space of language. It validates with a MuJoCo-CLEVR inspired environment and shows improved learning, generalization, and scalability over non-compositional abstractions and baselines.

ABSTRACT

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

Motivation & Objective

• Motivate and leverage language as a compositional abstraction for hierarchical RL to enable long-horizon skill learning.
• Develop a two-layer HAL framework where a high-level policy emits language instructions to a low-level, instruction-following policy.
• Create an open-source environment combining MuJoCo physics and CLEVR-inspired language for object manipulation tasks to study compositional generalization.
• Demonstrate that language-based abstractions enable better generalization and efficiency than non-compositional approaches and standard HRL baselines.

Proposed method

• Introduce HAL (Hierarchical Abstraction with Language), a two-layer HRL where the high-level policy outputs language instructions g and the low-level policy executes actions conditioned on s and g.
• Train the low-level language-conditioned policy with supervision from an instruction-relation function Psi(s, g) and a relabeling strategy called hindsight instruction relabeling (HIR) to densify rewards.
• Perform relabeling by associating trajectory segments with language goals drawn from the environment’s Omega(s) distribution, enabling learning from many language goals.
• Train the high-level policy to choose language instructions g from a structured subset I of G, effectively turning the problem into discrete-action RL over a vocabulary of instructions.
• Utilize a two-layer training regime where the low-level policy is trained with language as goal conditioning and the high-level policy learns to sequence instructions to achieve long-horizon tasks.
• Compare HAL against DDQN, HIRO, and Option-Critic on vision-based, long-horizon tasks and demonstrate transfer to image-based observations.

Experimental results

Research questions

• RQ1 How does language-as-abstraction compare to non-compositional representations for instruction following in HRL?
• RQ2 Can a high-level policy effectively compose a sequence of language instructions to solve long-horizon tasks?
• RQ3 Does the compositional structure of language enable systematic generalization to unseen instructions and settings?
• RQ4 How does HAL scale to vision-based observations and more diverse task suites?
• RQ5 How does HAL perform relative to standard HRL baselines on long-horizon manipulation tasks?

Key findings

• Language-based abstractions enable substantial learning and generalization gains over non-compositional representations and one-hot instruction encodings, especially as instruction sets scale.
• Hindsight instruction relabeling significantly densifies the learning signal for the low-level policy and is crucial for effective instruction-following learning.
• HAL learns to solve multiple high-level tasks with sparse rewards and transfers from state to pixel observations, outperforming DDQN, HIRO, and Option-Critic on standard tasks with high stability and lower variance.
• In vision-based settings, HAL maintains performance and sample efficiency, whereas baselines often fail to learn.
• Compositional language enables systematic generalization, with language exhibiting smaller generalization gaps and better zero-shot generalization to instructions unseen during training (relative to non-compositional baselines).
• The environment and code (HAL demo) are open-source and include continuous control tasks inspired by CLEVR for studying compositional, long-horizon manipulation.