[Paper Review] Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition

Ming Sun, Yuchen Yuan+2 more|arXiv (Cornell University)|Jun 14, 2018

Advanced Neural Network Applications17 citations

TL;DR

This paper proposes a novel end-to-end CNN framework with a one-squeeze multi-excitation (OSME) module and multi-attention multi-class constraint (MAMC) for fine-grained image recognition. It enables efficient, single-stage training by jointly localizing multiple discriminative parts and enforcing metric learning across attention regions, achieving state-of-the-art accuracy on CUB-200-2011, Stanford Dogs, Stanford Cars, and a new large-scale Dogs-in-the-Wild dataset without requiring part annotations.

ABSTRACT

Attention-based learning for fine-grained image recognition remains a challenging task, where most of the existing methods treat each object part in isolation, while neglecting the correlations among them. In addition, the multi-stage or multi-scale mechanisms involved make the existing methods less efficient and hard to be trained end-to-end. In this paper, we propose a novel attention-based convolutional neural network (CNN) which regulates multiple object parts among different input images. Our method first learns multiple attention region features of each input image through the one-squeeze multi-excitation (OSME) module, and then apply the multi-attention multi-class constraint (MAMC) in a metric learning framework. For each anchor feature, the MAMC functions by pulling same-attention same-class features closer, while pushing different-attention or different-class features away. Our method can be easily trained end-to-end, and is highly efficient which requires only one training stage. Moreover, we introduce Dogs-in-the-Wild, a comprehensive dog species dataset that surpasses similar existing datasets by category coverage, data volume and annotation quality. This dataset will be released upon acceptance to facilitate the research of fine-grained image recognition. Extensive experiments are conducted to show the substantial improvements of our method on four benchmark datasets.

Motivation & Objective

• To address the limitations of existing weakly-supervised fine-grained recognition methods that treat object parts in isolation and require complex, multi-stage training.
• To develop a lightweight, fully differentiable attention mechanism that localizes multiple non-correlated parts efficiently in one forward pass.
• To introduce a metric learning-based loss (MAMC) that enforces feature clustering by both attention region and class identity.
• To collect and release Dogs-in-the-Wild, a high-quality, large-scale dog species dataset to advance fine-grained recognition research.

Proposed method

• The one-squeeze multi-excitation (OSME) module applies channel-wise excitation to feature maps to identify multiple discriminative attention regions without explicit cropping or multiple forward passes.
• OSME uses a single global average pooling operation (one-squeeze) followed by multiple parallel excitation branches to generate diverse attention maps for different object parts.
• The multi-attention multi-class constraint (MAMC) loss pulls features from the same class and same attention region closer in embedding space while pushing features from different classes or different regions apart.
• MAMC is formulated as a metric learning objective that minimizes intra-class and intra-attention region distances while maximizing inter-class and inter-attention region distances.
• The entire network is trained end-to-end in a single stage, avoiding cascaded or multi-stage training procedures common in prior methods.
• The method is evaluated on four benchmark datasets, including the newly introduced Dogs-in-the-Wild, which features high category coverage, data volume, and annotation quality.

Experimental results

Research questions

• RQ1 Can a single, end-to-end trainable network effectively localize multiple discriminative parts in fine-grained images without part annotations?
• RQ2 Does enforcing correlation among multiple attention regions through a joint metric learning loss improve classification accuracy?
• RQ3 Can a lightweight, one-squeeze multi-excitation module outperform multi-stage or multi-scale attention mechanisms in efficiency and performance?
• RQ4 How does the proposed method compare to state-of-the-art methods on fine-grained recognition benchmarks, especially without relying on extra annotations?
• RQ5 To what extent does the newly introduced Dogs-in-the-Wild dataset present a more challenging benchmark for fine-grained recognition?

Key findings

• On the CUB-200-2011 dataset, the method achieves a top-1 accuracy of 87.1% with ResNet-101, matching the best result (MACNN) and surpassing PN-CNN (86.3%) and RAM (86.9%) without extra annotations.
• With ResNet-50, the method exceeds the second-best method without extra annotations (RAM) by 0.2% and the second-best with annotations (PN-CNN) by 0.8%.
• On Stanford Dogs, the method achieves 91.7% accuracy, outperforming all weakly-supervised methods except RACNN, which uses multiple training stages.
• On Stanford Cars, the method achieves 93.0% accuracy, surpassing all compared methods, including multi-scale and multi-stage models like DVAN and RAN.
• On the newly introduced Dogs-in-the-Wild dataset, the method achieves the highest accuracy, demonstrating its robustness on a more challenging, diverse, and realistic benchmark.
• Visualization confirms that OSME attention regions align with human-discriminative parts such as heads, tails, and body shapes, indicating effective and meaningful localization.