[Paper Review] GAMR: A Guided Attention Model for (visual) Reasoning

Mohit Vaishnav, T. Serre|arXiv (Cornell University)|Jun 10, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

GAMR proposes a guided attention mechanism that dynamically shifts focus across visual scenes using an LSTM controller to route task-relevant information to a memory bank, enabling robust, sample-efficient learning of visual reasoning tasks. It achieves state-of-the-art performance on SVRT and ART benchmarks and demonstrates strong zero-shot generalization by recombining learned operations.

ABSTRACT

Humans continue to outperform modern AI systems in their ability to flexibly parse and understand complex visual scenes. Here, we present a novel module for visual reasoning, the Guided Attention Model for (visual) Reasoning (GAMR), which instantiates an active vision theory -- positing that the brain solves complex visual reasoning problems dynamically -- via sequences of attention shifts to select and route task-relevant visual information into memory. Experiments on an array of visual reasoning tasks and datasets demonstrate GAMR's ability to learn visual routines in a robust and sample-efficient manner. In addition, GAMR is shown to be capable of zero-shot generalization on completely novel reasoning tasks. Overall, our work provides computational support for cognitive theories that postulate the need for a critical interplay between attention and memory to dynamically maintain and manipulate task-relevant visual information to solve complex visual reasoning tasks.

Motivation & Objective

• To develop a neural architecture that mimics active vision by dynamically shifting attention to task-relevant visual elements.
• To improve sample efficiency in visual reasoning by integrating memory routing with task-dependent attention.
• To enable zero-shot generalization across novel reasoning tasks by reusing previously learned elementary operations.
• To provide computational support for cognitive theories emphasizing the interplay between attention and memory in visual reasoning.
• To outperform existing state-of-the-art models on standard visual reasoning benchmarks such as SVRT and ART.

Proposed method

• The model uses a vision encoder (five convolutional blocks with instance normalization) to extract image representations.
• An LSTM-based controller generates internal queries at each time step to guide attention shifts based on current fixation state.
• A guided attention module selects and gates relevant visual features into a memory bank based on controller-generated queries.
• A relational reasoning module processes stored memory representations to infer relationships and solve reasoning tasks.
• The entire architecture is end-to-end trainable with a differentiable attention routing mechanism.
• Explainability methods are used to analyze attention patterns and validate compositional reasoning strategies.

Experimental results

Research questions

• RQ1 Can a neural network learn to perform dynamic, task-dependent attention shifts that improve visual reasoning performance?
• RQ2 Does integrating an internal controller with memory routing enhance sample efficiency in visual reasoning tasks?
• RQ3 Can the model generalize to novel reasoning tasks without fine-tuning, demonstrating zero-shot generalization?
• RQ4 How does the model’s attention behavior compare to human-like visual routines in complex relational tasks?
• RQ5 To what extent does the architecture maintain robustness under distribution shift and noise compared to baselines?

Key findings

• GAMR achieves state-of-the-art performance on both the Synthetic Visual Reasoning Test (SVRT) and the Abstract Reasoning Task (ART) benchmarks.
• The model demonstrates strong zero-shot generalization, transferring knowledge across tasks sharing abstract rules without retraining.
• GAMR maintains high accuracy under noise and distribution shifts, such as jittered shapes, while baselines like ESBN degrade significantly.
• The model learns to attend to relevant visual elements in a task-dependent manner, as validated by explainability analysis.
• GAMR achieves high performance with minimal data, learning complex relational rules in a data-efficient manner even with 500 training samples.
• Unlike object-centric models such as ESBN, GAMR successfully learns spatial relational rules even when shapes are separated into individual frames.