[Paper Review] Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions

Petros Christodoulou, Robert Tjarko Lange+2 more|arXiv (Cornell University)|Oct 7, 2019

Reinforcement Learning in Robotics19 references4 citations

TL;DR

This paper introduces Action Grammar Reinforcement Learning (AG-RL), a framework that uses grammar inference to discover hierarchical macro-actions from agent behavior, enhancing sample efficiency in deep reinforcement learning. By augmenting the action space with these learned macro-actions, AG-RL improves performance across 20 Atari games, achieving median gains of +96% and up to +3,756% in sample efficiency over SAC, and outperforms state-of-the-art methods in 17 of 20 games without extensive hyperparameter tuning.

ABSTRACT

From a young age humans learn to use grammatical principles to hierarchically combine words into sentences. Action grammars is the parallel idea, that there is an underlying set of rules (a "grammar") that govern how we hierarchically combine actions to form new, more complex actions. We introduce the Action Grammar Reinforcement Learning (AG-RL) framework which leverages the concept of action grammars to consistently improve the sample efficiency of Reinforcement Learning agents. AG-RL works by using a grammar inference algorithm to infer the "action grammar" of an agent midway through training. The agent's action space is then augmented with macro-actions identified by the grammar. We apply this framework to Double Deep Q-Learning (AG-DDQN) and a discrete action version of Soft Actor-Critic (AG-SAC) and find that it improves performance in 8 out of 8 tested Atari games (median +31%, max +668%) and 19 out of 20 tested Atari games (median +96%, maximum +3,756%) respectively without substantive hyperparameter tuning. We also show that AG-SAC beats the model-free state-of-the-art for sample efficiency in 17 out of the 20 tested Atari games (median +62%, maximum +13,140%), again without substantive hyperparameter tuning.

Motivation & Objective

• Address the critical challenge of poor sample efficiency in deep reinforcement learning, especially in environments with sparse rewards.
• Leverage the biological and cognitive parallels between language grammar and action hierarchies to improve learning efficiency and interpretability.
• Develop a method that automatically discovers and integrates structured, hierarchical macro-actions into the agent’s action space during training.
• Demonstrate that grammar-informed macro-actions significantly improve performance across diverse Atari environments without extensive hyperparameter tuning.
• Introduce generalizable techniques—Hindsight Action Replay and Abandon Ship—to enhance training stability and data efficiency when using long macro-actions.

Proposed method

• Pause training after a fixed number of steps to collect the agent’s action sequences and apply a grammar inference algorithm to identify recurring, structured action patterns.
• Transform identified action sequences into macro-actions (temporal abstractions) that represent higher-level behaviors, effectively composing primitive actions into complex, reusable units.
• Augment the agent’s original action space with these discovered macro-actions, enabling the agent to select them during continued training.
• Use Hindsight Action Replay (HAR) to improve data efficiency by reusing experiences from macro-actions, even when the original goal is not achieved.
• Implement the Abandon Ship technique to reduce variance during macro-action training by allowing early termination of long macro-actions if they fail, improving learning stability.
• Apply the framework to both DDQN and SAC agents (AG-DDQN and AG-SAC), using transfer learning to fine-tune networks rather than training from scratch.

Experimental results

Research questions

• RQ1 Can hierarchical action grammars discovered from agent behavior improve sample efficiency in deep reinforcement learning?
• RQ2 How do grammar-inferred macro-actions compare to hand-designed or repeated primitive actions in terms of performance and generalization across environments?
• RQ3 To what extent do the proposed techniques—Hindsight Action Replay and Abandon Ship—enhance learning stability and data efficiency when training with long macro-actions?
• RQ4 Does the AG-RL framework consistently outperform standard RL agents and state-of-the-art methods in sample efficiency across diverse Atari games?
• RQ5 Can the framework be applied effectively to both value-based (DDQN) and policy-based (SAC) RL algorithms without extensive hyperparameter tuning?

Key findings

• AG-RL improved performance in 8 out of 8 tested Atari games using AG-DDQN, with a median improvement of +31% and a maximum of +668% in sample efficiency over DDQN.
• AG-SAC achieved a median improvement of +96% and a maximum of +3,756% in sample efficiency across 19 out of 20 Atari games compared to SAC, with minimal hyperparameter tuning.
• AG-SAC outperformed the model-free state-of-the-art in 17 out of 20 Atari games in terms of sample efficiency, achieving a median improvement of +62% and a maximum of +13,140%.
• The Abandon Ship technique significantly reduced training variance, enabling effective learning with macro-actions exceeding 100 timesteps in length.
• Hindsight Action Replay was crucial for performance, as removing it caused AG-DDQN to perform no better than standard DDQN, indicating its role in maximizing experience reuse.
• On average, agents executed macro-actions of length 6.9 during evaluation (vs. 1.0 for primitive actions only), showing extensive use of higher-level actions, with macro-action lengths ranging from 2 to over 100 timesteps.