[Paper Review] Where to Focus: Deep Attention-based Spatially Recurrent Bilinear Networks for Fine-Grained Visual Recognition

Lin Wu, Yang Wang|arXiv (Cornell University)|Sep 18, 2017

Video Surveillance and Tracking Methods28 references6 citations

TL;DR

This paper proposes a deep attention-based spatially recurrent bilinear network that jointly detects discriminative object parts and extracts features through end-to-end training. By integrating soft attention with spatial LSTMs and bilinear pooling, the model dynamically focuses on critical regions, achieving state-of-the-art performance on fine-grained image classification and person re-identification benchmarks.

ABSTRACT

Fine-grained visual recognition typically depends on modeling subtle difference from object parts. However, these parts often exhibit dramatic visual variations such as occlusions, viewpoints, and spatial transformations, making it hard to detect. In this paper, we present a novel attention-based model to automatically, selectively and accurately focus on critical object regions with higher importance against appearance variations. Given an image, two different Convolutional Neural Networks (CNNs) are constructed, where the outputs of two CNNs are correlated through bilinear pooling to simultaneously focus on discriminative regions and extract relevant features. To capture spatial distributions among the local regions with visual attention, soft attention based spatial Long-Short Term Memory units (LSTMs) are incorporated to realize spatially recurrent yet visually selective over local input patterns. All the above intuitions equip our network with the following novel model: two-stream CNN layers, bilinear pooling layer, spatial recurrent layer with location attention are jointly trained via an end-to-end fashion to serve as the part detector and feature extractor, whereby relevant features are localized and extracted attentively. We show the significance of our network against two well-known visual recognition tasks: fine-grained image classification and person re-identification.

Motivation & Objective

• Address the challenge of fine-grained visual recognition where subtle differences between object categories are obscured by appearance variations such as occlusion, viewpoint changes, and lighting.
• Overcome limitations of standard bilinear pooling, which discards spatial relationships between features and fails to model spatial context.
• Improve feature representation by enabling dynamic, spatially aware attention that selectively focuses on salient object parts during feature extraction.
• Develop a deep learning framework that learns to attend to relevant regions end-to-end, mimicking human visual attention mechanisms.
• Enhance performance on fine-grained recognition tasks by combining bilinear feature pooling with spatial recurrence and attention mechanisms.

Proposed method

• Employ two-stream convolutional neural networks (CNNs) to extract features from the same image, enabling dual perspectives for improved discrimination.
• Apply bilinear pooling by computing the outer product of feature maps from the two CNN streams, capturing high-order feature interactions.
• Integrate spatial Long-Short Term Memory (LSTM) units to model spatial dependencies among local regions, enabling recurrent processing of feature maps.
• Incorporate soft attention mechanisms within the spatial LSTM to dynamically weight feature regions based on their relevance, focusing on discriminative parts.
• Use stochastic back-propagation to visualize attention maps, interpreting 'where' and 'what' the model focuses on during inference.
• Train the entire architecture end-to-end, jointly optimizing part detection, feature extraction, and attention weighting.

Experimental results

Research questions

• RQ1 How can a deep neural network dynamically and selectively attend to the most discriminative object parts in fine-grained visual recognition?
• RQ2 Can spatially recurrent LSTMs improve the modeling of spatial relationships in bilinear features, leading to better representation learning?
• RQ3 Does the integration of soft attention with bilinear pooling enhance performance on fine-grained classification and person re-identification tasks?
• RQ4 To what extent does the model’s attention mechanism align with human visual intuition in focusing on key object parts?
• RQ5 Can the proposed method outperform existing state-of-the-art models that rely on fixed spatial partitioning or hand-designed part detectors?

Key findings

• The proposed model achieves 93.4% top-1 accuracy on the Stanford Cars dataset, outperforming prior methods such as Krause et al. (92.6%) and Gosselin et al. (82.7%).
• On the CUB-200-2011 dataset, the model achieves state-of-the-art performance in fine-grained image classification, demonstrating superior feature localization and discrimination.
• In person re-identification, the model achieves higher rank-1 accuracy than multiple state-of-the-art approaches on VIPeR, CUHK03, and Market-1501 benchmarks.
• Visualization of attention maps confirms that the model learns to focus on semantically meaningful regions such as heads, wings, and body parts, aligning with human intuition.
• The model significantly outperforms multi-region bilinear models that rely on manual part partitioning, showing the advantage of end-to-end learned attention.
• The integration of spatial recurrence and attention leads to robustness against local distortions and spatial transformations, improving generalization on challenging visual recognition tasks.