[Paper Review] SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization

Hongbo Sun, Xiangteng He+1 more|arXiv (Cornell University)|Aug 31, 2022

Advanced Image and Video Retrieval Techniques9 citations

TL;DR

This paper proposes SIM-Trans, a vision transformer-based framework that enhances fine-grained visual categorization by modeling object structure through a Structure Information Learning (SIL) module and multi-level feature boosting (MFB). The SIL module leverages self-attention weights to capture spatial relationships among discriminative patches, while MFB integrates multi-level features and contrastive learning to improve feature robustness, achieving state-of-the-art performance on CUB-200-2011 and iNaturalist 2017 benchmarks.

ABSTRACT

Fine-grained visual categorization (FGVC) aims at recognizing objects from similar subordinate categories, which is challenging and practical for human's accurate automatic recognition needs. Most FGVC approaches focus on the attention mechanism research for discriminative regions mining while neglecting their interdependencies and composed holistic object structure, which are essential for model's discriminative information localization and understanding ability. To address the above limitations, we propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning to contain both the appearance information and structure information. Specifically, we encode the image into a sequence of patch tokens and build a strong vision transformer framework with two well-designed modules: (i) the structure information learning (SIL) module is proposed to mine the spatial context relation of significant patches within the object extent with the help of the transformer's self-attention weights, which is further injected into the model for importing structure information; (ii) the multi-level feature boosting (MFB) module is introduced to exploit the complementary of multi-level features and contrastive learning among classes to enhance feature robustness for accurate recognition. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily, which only depends on the attention weights that come with the vision transformer itself. Extensive experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks. The code is available at https://github.com/PKU-ICST-MIPL/SIM-Trans_ACMMM2022.

Motivation & Objective

• To address the limitation of existing fine-grained visual categorization methods that overlook interdependencies among discriminative regions and holistic object structure.
• To improve discriminative representation learning by integrating both appearance and structural information into vision transformers.
• To develop a lightweight, plug-in module that enhances feature localization and robustness without requiring additional annotations.
• To achieve state-of-the-art performance on large-scale and fine-grained benchmarks, including iNaturalist 2017 with over 95,000 test images.

Proposed method

• The Structure Information Learning (SIL) module mines spatial context relations among significant patches within an object using self-attention weights from the vision transformer, injecting structural awareness into the model.
• The SIL module is applied at multiple layers of the transformer, with optimal performance achieved when injected into three layers, enhancing spatial reasoning and attention to discriminative regions.
• The Multi-level Feature Boosting (MFB) module fuses features across different transformer layers to exploit complementary representations and improve robustness.
• MFB incorporates contrastive learning with a learnable temperature hyper-parameter α to emphasize hard negative samples and enhance feature discrimination.
• The two modules are lightweight and can be seamlessly plugged into any vision transformer architecture for end-to-end training using only the attention weights inherent to the transformer.
• The model is trained end-to-end with cross-entropy loss and contrastive loss, enabling joint optimization of classification and feature representation.

Experimental results

Research questions

• RQ1 How can object structure information be effectively modeled in vision transformers for fine-grained visual categorization?
• RQ2 Can self-attention weights be leveraged to capture spatial dependencies among discriminative patches without additional supervision?
• RQ3 Does multi-level feature fusion combined with contrastive learning improve feature robustness and classification accuracy in fine-grained recognition?
• RQ4 What is the optimal depth and configuration for injecting structure-aware modules into a vision transformer?
• RQ5 Can a plug-in module that relies only on attention weights achieve state-of-the-art performance on large-scale fine-grained benchmarks?

Key findings

• The proposed SIM-Trans achieves 91.8% top-1 accuracy on the CUB-200-2011 benchmark, outperforming the baseline by 1.2 percentage points.
• The Structure Information Learning (SIL) module alone improves accuracy by 0.5% over the baseline, demonstrating its effectiveness in enhancing structural understanding.
• Adding the MFB module with contrastive learning further boosts performance by 0.4%, highlighting the benefit of hard negative mining and feature discrimination.
• The ablation study shows that injecting SIL into three transformer layers yields the best performance, indicating an optimal trade-off between receptive field and localization precision.
• On the large-scale iNaturalist 2017 benchmark with over 95,000 test images and 5,000+ categories, SIM-Trans achieves state-of-the-art performance, confirming its scalability and generalization ability.
• Hyperparameter analysis reveals that α = 0.3 in the MFB module yields the best performance, as higher values degrade results by including easy negatives in contrastive learning.