[Paper Review] Few-shot Fine-grained Image Classification via Multi-Frequency Neighborhood and Double-cross Modulation

Hegui Zhu, Zhan Gao+3 more|arXiv (Cornell University)|Jul 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes FicNet, a few-shot fine-grained image classification model using Multi-Frequency Neighborhood (MFN) and Double-cross Modulation (DCM) to enhance feature representation and alignment. MFN captures spatial and frequency-domain structural patterns to reduce intra-class variation, while DCM uses bi-crisscross and 3D cross-attention to modulate features with global context and task-specific attention, achieving 93.17% accuracy on CUB-200 and 95.36% on Stanford Cars—surpassing general fine-grained methods.

ABSTRACT

Traditional fine-grained image classification typically relies on large-scale training samples with annotated ground-truth. However, some sub-categories have few available samples in real-world applications, and current few-shot models still have difficulty in distinguishing subtle differences among fine-grained categories. To solve this challenge, we propose a novel few-shot fine-grained image classification network (FicNet) using multi-frequency neighborhood (MFN) and double-cross modulation (DCM). MFN focuses on both spatial domain and frequency domain to capture multi-frequency structural representations, which reduces the influence of appearance and background changes to the intra-class distance. DCM consists of bi-crisscross component and double 3D cross-attention component. It modulates the representations by considering global context information and inter-class relationship respectively, which enables the support and query samples respond to the same parts and accurately identify the subtle inter-class differences. The comprehensive experiments on three fine-grained benchmark datasets for two few-shot tasks verify that FicNet has excellent performance compared to the state-of-the-art methods. Especially, the experiments on two datasets, "Caltech-UCSD Birds" and "Stanford Cars", can obtain classification accuracy 93.17\% and 95.36\%, respectively. They are even higher than that the general fine-grained image classification methods can achieve.

Motivation & Objective

• Address the challenge of few-shot fine-grained image classification where sub-categories have minimal training samples and subtle visual differences.
• Reduce intra-class variation caused by appearance and background changes in real-world data.
• Improve inter-class discrimination by focusing on local, discriminative parts across support and query images.
• Enable accurate classification without part annotations or bounding boxes by learning robust, aligned representations.
• Balance model performance with computational efficiency through structured attention and frequency-aware feature learning.

Proposed method

• Multi-Frequency Neighborhood (MFN) decomposes feature maps into multi-frequency components using spectral analysis to extract compact, smooth structural patterns.
• MFN enhances robustness by leveraging self-similarity in neighborhoods to suppress irrelevant appearance and background variations.
• Double-cross Modulation (DCM) combines bi-crisscross (BCC) and double 3D cross-attention (DCA) to modulate features using global context and task-specific relationships.
• BCC progressively aggregates context from all directions using crisscross convolutions, improving feature completeness.
• DCA uses 3D double-cross convolutions to refine 4D cross-correlation tensors, preserving geometric detail while focusing on discriminative regions.
• The DCM module enables cross-attention between support and query features, aligning responses to the same object parts despite viewpoint or occlusion differences.

Experimental results

Research questions

• RQ1 Can multi-frequency feature representation reduce intra-class variation caused by background and appearance changes in few-shot fine-grained classification?
• RQ2 How can global context and task-specific attention be jointly modeled to improve feature alignment between support and query images?
• RQ3 Does 3D double-cross attention outperform standard cross-attention mechanisms in preserving spatial detail and improving discrimination?
• RQ4 To what extent does bi-crisscross context aggregation enhance feature completeness and classification accuracy?
• RQ5 Can the proposed method achieve state-of-the-art performance without requiring part-level annotations?

Key findings

• FicNet achieved 93.17% accuracy on the CUB-200 dataset in the 5-shot setting, exceeding the performance of general fine-grained classification methods.
• On the Stanford Cars dataset, FicNet reached 95.36% accuracy in the 5-shot setting, demonstrating strong generalization on complex vehicle categories.
• The ablation study confirmed that combining BCC and DCA in DCM improved accuracy by 2.06% and 2.3% over the baseline for 1-shot and 5-shot tasks, respectively.
• Using two crisscross operations (L=2) in BCC provided optimal performance with minimal memory and time overhead, balancing efficiency and accuracy.
• Replacing standard cross-attention modules like CAN and CCA with DCA improved accuracy by 0.71% and 0.34% on CUB-200 and mini-ImageNet, respectively.
• The full FicNet model with MFN, BCC, and DCA achieved 80.97% accuracy on CUB-200 (1-shot) and 93.17% on 5-shot, outperforming state-of-the-art methods.