[Paper Review] Strategic Attentive Writer for Learning Macro-Actions

Alexander -, Vezhnevets+6 more|arXiv (Cornell University)|Jun 15, 2016

Artificial Intelligence in Games20 references80 citations

TL;DR

STRAW is a deep recurrent neural network that learns temporally abstracted macro-actions end-to-end in reinforcement learning by maintaining and updating a multi-step action plan, with replanning decisions and action sequences learned from environment rewards. It improves performance on Atari games requiring long-term planning, such as Ms. Pacman and Frostbite, by enabling structured exploration and efficient computation through implicit macro-action discovery.

ABSTRACT

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.

Motivation & Objective

• To develop a deep learning architecture that learns high-level, temporally extended actions (macro-actions) from raw environment interactions without prior knowledge or handcrafted subgoals.
• To enable structured exploration in reinforcement learning by learning when to commit to a plan and when to re-plan based on environmental feedback.
• To reduce computational cost by deferring observation processing during plan execution, freeing resources during committed phases.
• To generalize across sequence tasks, including both control (Atari games) and sequence prediction (text), by learning frequent n-grams as macro-actions.
• To demonstrate end-to-end learning of macro-actions through a novel architecture that implicitly captures temporal abstractions from reward signals alone.

Proposed method

• STRAW uses a dual-module recurrent architecture: a feature extractor (e.g., CNN) processes observations, and a planning module generates and updates a multi-step action plan.
• The model maintains a commitment plan (c_t) that determines how long the current action sequence is committed to before re-planning.
• A noisy communication channel between the feature extractor and planning module injects structured randomness, promoting exploration over extended time horizons.
• Action plans are generated via an attention mechanism that attends to relevant parts of the internal state to produce sequences of actions.
• The network is trained via policy gradient methods using only the environment’s sparse reward signal, with no pseudo-rewards or subgoal supervision.
• The architecture supports both reinforcement learning and supervised sequence prediction, with macro-actions emerging naturally as frequent action patterns.

Experimental results

Research questions

• RQ1 Can a deep recurrent network learn temporally abstracted macro-actions end-to-end from only sparse rewards, without handcrafted subgoals or pseudo-rewards?
• RQ2 How does learning to commit to action plans improve exploration and sample efficiency in reinforcement learning?
• RQ3 Can the model adapt its planning horizon dynamically based on environmental conditions, such as obstacles or threats?
• RQ4 Does the proposed architecture generalize beyond control tasks to other sequence modeling problems, such as text prediction?
• RQ5 What is the impact of structured exploration via noisy communication on learning performance and macro-action discovery?

Key findings

• STRAW achieves substantial performance improvements on Atari games requiring long-term planning, such as Ms. Pacman and Frostbite, outperforming standard baselines and even LSTM models in several cases.
• On Frostbite, STRAW learns macro-actions corresponding to meaningful high-level behaviors, such as jumping between ice floes and collecting fish, as visualized in the planning outputs.
• In Amidar, STRAW dynamically adjusts its planning horizon: it shortens macro-actions when enemies are near and resumes longer plans when the path is clear, demonstrating adaptive re-planning.
• The model learns to commit to plans for extended durations during training, with commitment levels stabilizing after ~200 epochs on most games, except Breakout, where frequent re-planning is optimal due to high-precision demands.
• Ablative studies show that learning when to re-plan is more effective than fixed or always-replanning strategies, and that higher-dimensional noise channels improve performance, though with diminishing returns.
• When trained on text prediction, STRAW naturally learns to generate frequent n-grams as macro-actions, confirming the architecture’s generality across sequence tasks.