[Paper Review] Deep Attention Recurrent Q-Network

Ivan Sorokin, А. А. Селезнев+3 more|arXiv (Cornell University)|Dec 5, 2015

Reinforcement Learning in Robotics15 references100 citations

TL;DR

This paper proposes Deep Attention Recurrent Q-Network (DARQN), an extension of Deep Q-Network (DQN) that integrates soft and hard attention mechanisms with Long Short-Term Memory (LSTM) networks to improve decision-making in Atari 2600 games. By enabling the agent to focus on relevant visual regions, DARQN achieves superior performance over DQN and DRQN on select games while providing interpretable attention visualization for online monitoring of learning behavior.

ABSTRACT

A deep learning approach to reinforcement learning led to a general learner able to train on visual input to play a variety of arcade games at the human and superhuman levels. Its creators at the Google DeepMind's team called the approach: Deep Q-Network (DQN). We present an extension of DQN by "soft" and "hard" attention mechanisms. Tests of the proposed Deep Attention Recurrent Q-Network (DARQN) algorithm on multiple Atari 2600 games show level of performance superior to that of DQN. Moreover, built-in attention mechanisms allow a direct online monitoring of the training process by highlighting the regions of the game screen the agent is focusing on when making decisions.

Motivation & Objective

• To improve the generalization and learning efficiency of Deep Q-Network (DQN) in reinforcement learning for Atari 2600 games.
• To address DQN’s limitations in long-term memory and high training time by integrating recurrent and attention mechanisms.
• To enhance interpretability of agent decisions by enabling visualization of attention regions during gameplay.
• To evaluate the effectiveness of both soft and hard attention mechanisms in a recurrent Q-network architecture.
• To explore whether attention mechanisms reduce computational load and improve training speed without sacrificing performance.

Proposed method

• Integrates a convolutional neural network (CNN) to extract feature maps from game frames at each time step.
• Employs an attention network to compute soft or hard attention weights over spatial regions of the feature maps, producing a context vector.
• Uses a Long Short-Term Memory (LSTM) network to maintain hidden and memory states, integrating context vectors across time steps.
• For soft attention, computes a weighted sum of feature vectors using a differentiable attention mechanism based on a multi-layer perceptron and softmax.
• For hard attention, uses a stochastic sampling mechanism with policy gradient training to select a single region of interest.
• Trains the entire model end-to-end using backpropagation through time with RMSProp and an $ε$-greedy exploration strategy.

Experimental results

Research questions

• RQ1 Can the integration of attention mechanisms into DQN improve performance on Atari 2600 games compared to standard DQN and DRQN?
• RQ2 Does the use of attention mechanisms enhance interpretability by enabling visualization of the agent’s focus during decision-making?
• RQ3 How do soft and hard attention mechanisms compare in terms of learning stability and performance on different Atari games?
• RQ4 Can attention mechanisms reduce the number of parameters and computational cost in deep reinforcement learning models?
• RQ5 Does the attention mechanism improve long-term memory and temporal generalization beyond the four-frame context used in standard DQN?

Key findings

• On Seaquest, both soft and hard attention-based DARQN models outperformed DQN and DRQN, with the soft attention variant achieving superior results.
• The soft attention mechanism successfully visualized the agent focusing on the ball trajectory in Breakout and on the oxygen gauge and enemies in Seaquest.
• The hard attention mechanism failed to learn the resurfacing behavior in Seaquest, likely due to local optima in policy gradient training.
• In Breakout, neither soft nor hard DARQN models surpassed the performance of the original DQN, suggesting limitations in unroll depth or training stability.
• Attention visualization confirmed that the agent dynamically shifts focus between relevant game elements, such as the ball or enemies, providing interpretability into decision-making.
• The attention mechanism reduced the effective input size by focusing on salient regions, suggesting potential for computational efficiency, though not explicitly quantified in the paper.