[Paper Review] Deep Reinforcement Learning With Macro-Actions

Ishan Durugkar, Clemens Rosenbaum+2 more|arXiv (Cornell University)|Jun 15, 2016

Reinforcement Learning in Robotics15 references11 citations

TL;DR

This paper proposes integrating macro-actions—predefined sequences of primitive actions—into Deep Q-Networks (DQN) to accelerate learning and improve performance in Atari 2600 games. By enabling the agent to take temporally extended actions, the method reduces reliance on sparse reward signals, increases Q-value confidence, and leads to faster convergence and superior final scores in six out of seven games tested.

ABSTRACT

Deep reinforcement learning has been shown to be a powerful framework for learning policies from complex high-dimensional sensory inputs to actions in complex tasks, such as the Atari domain. In this paper, we explore output representation modeling in the form of temporal abstraction to improve convergence and reliability of deep reinforcement learning approaches. We concentrate on macro-actions, and evaluate these on different Atari 2600 games, where we show that they yield significant improvements in learning speed. Additionally, we show that they can even achieve better scores than DQN. We offer analysis and explanation for both convergence and final results, revealing a problem deep RL approaches have with sparse reward signals.

Motivation & Objective

• Improve the sample efficiency and convergence speed of deep reinforcement learning agents in high-dimensional control tasks.
• Address the challenge of sparse reward signals in Atari environments, where feedback is delayed and infrequent.
• Investigate whether temporal abstraction via macro-actions can lead to more reliable policy learning than primitive actions alone.
• Explore how macro-actions interact with deep neural networks to improve state representation learning and Q-value estimation.
• Demonstrate that macro-actions can yield better final performance than standard DQN, even when the agent has access to atomic actions.

Proposed method

• Extend the DQN architecture to include macro-actions as additional, atomic-like actions in the action space.
• Define macros as fixed, deterministic sequences of primitive actions (e.g., 'move left 5 times, then jump').
• Train the DQN agent using a replay buffer that includes transitions from both primitive actions and macro-actions.
• Use standard DQN training with experience replay and target networks, while allowing the agent to select macros as single actions.
• Construct macros via simple heuristics: repeated action sequences (e.g., 'left, left, left') and fixed-length action sequences.
• Evaluate the impact of macro-action selection on learning speed, final performance, and Q-value confidence.

Experimental results

Research questions

• RQ1 Can macro-actions significantly reduce training time in deep reinforcement learning on Atari 2600 games?
• RQ2 Do macro-actions lead to better final performance compared to standard DQN with only primitive actions?
• RQ3 How do macro-actions affect the confidence and stability of Q-value estimates in deep Q-networks?
• RQ4 Why do macro-actions improve learning in environments with sparse rewards, despite reducing action space granularity?
• RQ5 To what extent do different macro construction strategies (e.g., repeated actions, fixed sequences) influence learning outcomes?

Key findings

• Macro-actions reduced training time by approximately 50% compared to standard DQN, achieving comparable performance in 50 million steps versus 100 million steps.
• The DQN agent with macro-actions achieved higher final scores than standard DQN in six out of seven Atari 2600 games tested.
• Q-value confidence, measured as the difference between the best and second-best Q-values, was orders of magnitude higher when using macro-actions, reducing the impact of approximation error.
• The improved Q-value spread due to macro-actions led to more reliable greedy action selection, effectively increasing the action gap.
• The deep neural network learned state representations more efficiently with macro-actions, as larger state changes were more salient and easier to encode.
• In games like Ms. Pac-Man, simple repeated-action macros outperformed more complex strategies, suggesting that macro design must align with game mechanics.