[Paper Review] Recurrent Models of Visual Attention

Volodymyr Mnih, Nicolas Heess+2 more|arXiv (Cornell University)|Jun 24, 2014

Visual Attention and Saliency Detection22 references2,273 citations

TL;DR

This paper proposes a recurrent neural network model that selectively attends to high-resolution regions of images or videos, reducing computational cost by processing only relevant areas. Using reinforcement learning, the model learns task-specific attention policies, outperforming CNNs on cluttered image classification and learning object tracking without explicit supervision.

ABSTRACT

Applying convolutional neural networks to large images is computationally ex-pensive because the amount of computation scales linearly with the number of image pixels. We present a novel recurrent neural network model that is ca-pable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Like convolutional neural networks, the proposed model has a degree of translation invariance built-in, but the amount of computation it per-forms can be controlled independently of the input image size. While the model is non-differentiable, it can be trained using reinforcement learning methods to learn task-specific policies. We evaluate our model on several image classification tasks, where it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem, where it learns to track a simple object without an explicit training signal for doing so. 1

Motivation & Objective

• To address the high computational cost of processing large images with convolutional neural networks (CNNs).
• To develop a model that performs computation proportional to the relevant image regions rather than the full image size.
• To enable translation-invariant feature learning through adaptive attention mechanisms.
• To train the model using reinforcement learning for task-specific attention policies.

Proposed method

• The model uses a recurrent network to predict a sequence of image regions to attend to, based on hidden state and image features.
• Only selected regions are processed at high resolution, while the rest are ignored, reducing computational load.
• The attention mechanism is non-differentiable, so policy gradients from reinforcement learning are used to train the network.
• The model maintains translation invariance through shared weights in the recurrent structure and local processing.
• A reward signal from the task guides the attention policy via policy gradient methods.
• The architecture supports both static image classification and dynamic visual control tasks.

Experimental results

Research questions

• RQ1 Can a recurrent model learn to attend to relevant image regions to reduce computation without sacrificing performance?
• RQ2 How does selective attention improve classification accuracy on cluttered images compared to full-image processing?
• RQ3 Can the model learn to track moving objects without explicit supervision or ground-truth tracking signals?
• RQ4 To what extent does the model's performance scale with input image size compared to standard CNNs?
• RQ5 How effective is reinforcement learning in training a non-differentiable attention mechanism for visual tasks?

Key findings

• The model significantly outperforms a CNN baseline on image classification tasks with cluttered images, demonstrating improved robustness to distractors.
• The model achieves high performance with substantially reduced computation, as it processes only a fraction of the image pixels.
• On a dynamic visual control task, the model learns to track a moving object without any explicit training signal for tracking.
• The attention mechanism learns to focus on relevant features, such as object boundaries or key parts, improving generalization.
• The model's performance remains stable across varying input sizes, unlike CNNs whose computation scales linearly with image size.
• Reinforcement learning effectively trains the non-differentiable attention policy, enabling task-specific adaptation.