[Paper Review] Learning Granularity-Aware Convolutional Neural Network for Fine-Grained Visual Classification

Jianwei Song, Ruoyu Yang|arXiv (Cornell University)|Mar 4, 2021

Advanced Neural Network Applications24 references4 citations

TL;DR

This paper proposes Granularity-Aware Convolutional Neural Network (GA-CNN), a weakly supervised method for fine-grained visual classification that learns multi-granularity features by leveraging feature maps from multiple stages of a CNN backbone. By progressively refining discriminative part localization through stage-specific classifiers and an object-attentive module, GA-CNN achieves state-of-the-art accuracy on CUB-200-2011, FGVC-Aircraft, and Stanford Cars without requiring bounding box or part annotations.

ABSTRACT

Locating discriminative parts plays a key role in fine-grained visual classification due to the high similarities between different objects. Recent works based on convolutional neural networks utilize the feature maps taken from the last convolutional layer to mine discriminative regions. However, the last convolutional layer tends to focus on the whole object due to the large receptive field, which leads to a reduced ability to spot the differences. To address this issue, we propose a novel Granularity-Aware Convolutional Neural Network (GA-CNN) that progressively explores discriminative features. Specifically, GA-CNN utilizes the differences of the receptive fields at different layers to learn multi-granularity features, and it exploits larger granularity information based on the smaller granularity information found at the previous stages. To further boost the performance, we introduce an object-attentive module that can effectively localize the object given a raw image. GA-CNN does not need bounding boxes/part annotations and can be trained end-to-end. Extensive experimental results show that our approach achieves state-of-the-art performances on three benchmark datasets.

Motivation & Objective

• Address the challenge of distinguishing subtle visual differences in fine-grained categories where traditional CNNs fail to localize discriminative parts due to large receptive fields.
• Overcome the limitation of relying on the last convolutional layer—which focuses on whole objects—by exploiting smaller receptive fields in earlier layers to capture part-level details.
• Develop a weakly supervised approach that does not require bounding box or part-level annotations, enabling end-to-end training.
• Improve feature representation by progressively combining multi-granularity features from different network stages, enhancing discriminative power.
• Introduce an object-attentive module to localize the object region using attention maps, further boosting performance without supervision.

Proposed method

• Utilize feature maps from multiple stages (L−S+1 to L) of a ResNet-50 backbone to learn multi-granularity features, with increasing receptive field size from early to late layers.
• Apply 1×1 and 3×3 convolutional blocks followed by global top-k pooling (GTKP) to each stage’s feature map to extract compact, discriminative representations.
• Train a separate classifier for each stage’s pooled features using cross-entropy loss, enabling progressive learning of discriminative parts from coarse to fine granularity.
• Integrate an object-attentive module that generates an attention map to estimate the object’s bounding box from the raw image, improving localization and feature quality.
• Combine predictions from all stage-specific classifiers via a weighted average (with α=0.3), fusing coarse and fine-grained predictions for final classification.
• Optimize the entire network end-to-end using SGD with cosine annealing learning rate scheduling and weight decay, with different initial learning rates for backbone and newly added layers.

Experimental results

Research questions

• RQ1 Can multi-granularity feature learning from different network stages improve fine-grained visual classification beyond relying on the final convolutional layer?
• RQ2 How effective is an object-attentive module in localizing the object region without bounding box annotations?
• RQ3 Does progressive refinement of features from coarse to fine granularity enhance discriminative representation learning?
• RQ4 How does the choice of pooling operation (GAP, GMP, GTKP) affect performance in weakly supervised FGVC?
• RQ5 To what extent is the model’s performance sensitive to the hyperparameter α in the weighted fusion of multi-granularity predictions?

Key findings

• GA-CNN achieves state-of-the-art accuracy of 90.4% on the CUB-200-2011 dataset using ResNet-50, surpassing all prior weakly supervised methods even with more advanced backbones.
• On FGVC-Aircraft, GA-CNN achieves the best performance among methods using the same ResNet-50 backbone, demonstrating strong generalization.
• The ablation study shows that adding the object-attentive module (OAM) improves accuracy by 1.7% over the baseline with only granularity-specific classifiers (from 88.7% to 90.4%).
• The global top-k pooling (GTKP) method outperforms both GAP and GMP, achieving 90.38% accuracy, indicating that capturing multiple discriminative parts is more effective than relying on the maximum or average activation.
• The model is robust to the threshold hyperparameter α, with accuracy varying by less than 0.7% across α values from 0.1 to 0.5, showing stable performance under different fusion weights.
• Visualization confirms that the object-attentive module effectively estimates the object region, with the predicted bounding boxes closely aligning with ground-truth in most cases.