[Paper Review] Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification

Jijie Wu, Dongliang Chang+6 more|arXiv (Cornell University)|Nov 30, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a bi-directional feature reconstruction network (Bi-FRN) for fine-grained few-shot image classification, introducing a mutual reconstruction mechanism that simultaneously increases inter-class variations (via support-to-query reconstruction) and reduces intra-class variations (via query-to-support reconstruction). The method achieves state-of-the-art performance across three benchmark datasets, outperforming prior reconstruction-based and metric-based approaches by learning more discriminative, fine-grained features through dual-directional feature reconstruction and a self-reconstruction module.

ABSTRACT

The main challenge for fine-grained few-shot image classification is to learn feature representations with higher inter-class and lower intra-class variations, with a mere few labelled samples. Conventional few-shot learning methods however cannot be naively adopted for this fine-grained setting -- a quick pilot study reveals that they in fact push for the opposite (i.e., lower inter-class variations and higher intra-class variations). To alleviate this problem, prior works predominately use a support set to reconstruct the query image and then utilize metric learning to determine its category. Upon careful inspection, we further reveal that such unidirectional reconstruction methods only help to increase inter-class variations and are not effective in tackling intra-class variations. In this paper, we for the first time introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. In addition to using the support set to reconstruct the query set for increasing inter-class variations, we further use the query set to reconstruct the support set for reducing intra-class variations. This design effectively helps the model to explore more subtle and discriminative features which is key for the fine-grained problem in hand. Furthermore, we also construct a self-reconstruction module to work alongside the bi-directional module to make the features even more discriminative. Experimental results on three widely used fine-grained image classification datasets consistently show considerable improvements compared with other methods. Codes are available at: https://github.com/PRIS-CV/Bi-FRN.

Motivation & Objective

• To address the limitation of existing few-shot learning methods in fine-grained image classification, where models inadvertently increase intra-class variation and reduce inter-class variation.
• To overcome the shortcomings of unidirectional reconstruction methods that only increase inter-class variations without effectively reducing intra-class variations.
• To propose a novel bi-directional reconstruction mechanism that simultaneously enhances inter-class discrimination and suppresses intra-class variability in fine-grained features.
• To improve feature discriminability by integrating a self-reconstruction module alongside the mutual reconstruction module for more robust representation learning.

Proposed method

• The model employs a bi-directional feature reconstruction mechanism: reconstructing query features from support features to increase inter-class variations, and reconstructing support features from query features to reduce intra-class variations.
• A Feature Mutual Reconstruction Module (FMRM) is introduced, which jointly optimizes both reconstruction directions using a weighted loss function combining reconstruction errors from both directions.
• A self-reconstruction module (FSRM) is added to further enhance feature discriminability by reconstructing features from their own representations.
• The framework uses a feature extraction backbone (e.g., Conv-4 or ResNet-12), followed by metric learning for classification, with reconstruction losses applied at the feature level.
• The reconstruction process uses a learned decoder (inverse ResNet) to recover images from features, enabling visualization and validation of reconstruction fidelity.
• The overall training objective combines metric learning loss and dual reconstruction losses, with hyperparameters to balance inter- and intra-class learning dynamics.

Experimental results

Research questions

• RQ1 Can a bi-directional reconstruction mechanism effectively reduce intra-class variation while increasing inter-class variation in fine-grained few-shot learning?
• RQ2 How does query-to-support reconstruction compare to traditional support-to-query reconstruction in improving feature discriminability?
• RQ3 To what extent does the inclusion of a self-reconstruction module enhance feature quality in the absence of strong supervision?
• RQ4 Does the mutual reconstruction mechanism lead to more localized and semantically consistent attention maps compared to unidirectional or baseline methods?
• RQ5 How does the proposed method perform across diverse fine-grained datasets under both 1-shot and 5-shot settings?

Key findings

• The proposed Bi-FRN method achieves state-of-the-art accuracy on all three benchmark datasets: CUB-200-211, Stanford Dogs, and FG-WILLOW, outperforming prior methods including FRN and ProtoNet.
• On the CUB-200-211 dataset, Bi-FRN achieves 86.7% accuracy under the 5-way 1-shot setting and 92.1% under the 5-way 5-shot setting using ResNet-12.
• The ablation study confirms that both the FSRM and FMRM modules are essential, with performance dropping significantly when either is removed, demonstrating their complementary roles.
• The mutual reconstruction module (FMRM) outperforms unidirectional variants (S→Q or Q→S alone), proving that bidirectional learning is more effective for fine-grained feature learning.
• Visualization results show that reconstructed features from same-class samples are semantically consistent, while those from different classes are dissimilar, confirming the model's ability to learn robust, class-aligned representations.
• The decoder-based image recovery experiment validates that the reconstructed features preserve structural and semantic details, especially when reconstructing same-class samples.