[Paper Review] Feature Fusion Vision Transformer for Fine-Grained Visual Categorization

Jun Wang, Xiaohan Yu+1 more|arXiv (Cornell University)|Jul 6, 2021

Advanced Neural Network Applications38 references84 citations

TL;DR

FFVT extends ViT with a feature fusion mechanism and a Mutual Attention Weight Selection (MAWS) to aggregate local, mid-level, and high-level tokens for FGVC, achieving state-of-the-art results on four FGVC benchmarks.

ABSTRACT

The core for tackling the fine-grained visual categorization (FGVC) is to learn subtle yet discriminative features. Most previous works achieve this by explicitly selecting the discriminative parts or integrating the attention mechanism via CNN-based approaches.However, these methods enhance the computational complexity and make the modeldominated by the regions containing the most of the objects. Recently, vision trans-former (ViT) has achieved SOTA performance on general image recognition tasks. Theself-attention mechanism aggregates and weights the information from all patches to the classification token, making it perfectly suitable for FGVC. Nonetheless, the classifi-cation token in the deep layer pays more attention to the global information, lacking the local and low-level features that are essential for FGVC. In this work, we proposea novel pure transformer-based framework Feature Fusion Vision Transformer (FFVT)where we aggregate the important tokens from each transformer layer to compensate thelocal, low-level and middle-level information. We design a novel token selection mod-ule called mutual attention weight selection (MAWS) to guide the network effectively and efficiently towards selecting discriminative tokens without introducing extra param-eters. We verify the effectiveness of FFVT on three benchmarks where FFVT achieves the state-of-the-art performance.

Motivation & Objective

• Address the need for discriminative local features in fine-grained visual categorization (FGVC).
• Leverage vision transformers to capture both global and local information without CNN biases.
• Develop a token selection mechanism to pick informative tokens across layers without extra parameters.
• Fuse multi-level tokens to enrich the final classifier with local and mid/high-level information.
• Validate on four FGVC benchmarks, including small-scale and ultra-fine-grained datasets.

Proposed method

• Use a pure ViT backbone to process images split into patches with a class token for classification.
• Introduce Feature Fusion Module to replace inputs of the last transformer layer with selected tokens from previous layers (excluding the class token).
• Propose Mutual Attention Weight Selection (MAWS) to select discriminative tokens for fusion using self-attention scores.
• Compute mutual attention weights by normalizing attention scores from the classification token side and the token-side context, then select top-K tokens per layer.
• Aggregate K local/mid/high-level tokens from each layer as z_local and fuse them with the last layer inputs as z_ff, feeding into the final classifier.
• Maintain no extra learnable parameters in MAWS; rely on attention-derived signals for token selection.

Experimental results

Research questions

• RQ1 Can a pure transformer architecture, with multi-layer token fusion, achieve competitive FGVC performance on both small-scale and ultra-fine-grained datasets?
• RQ2 Does selectively aggregating local and mid-level tokens across layers improve FGVC performance compared to using only final-layer class-token information?
• RQ3 Is the MAWS-based token selection effective and efficient without adding learnable parameters?

Key findings

• FFVT achieves state-of-the-art results on four FGVC benchmarks, surpassing many CNN-based approaches.
• On CUB-200-2011, FFVT with ViT-B_16 attains 91.6% accuracy, closely following TransFG at 91.7%.
• On Stanford Dogs, FFVT reaches 91.5% accuracy, outperforming the second-best TransFG by 0.9%.
• On CottonCultivar80, FFVT achieves 57.92% accuracy, the highest among reported methods.
• On SoyCultivarLocal, FFVT achieves 44.17% accuracy, higher than prior methods.