[Paper Review] Object-aware Long-short-range Spatial Alignment for Few-Shot Fine-Grained Image Classification

Yike Wu, Bo Zhang+5 more|arXiv (Cornell University)|Aug 30, 2021

Domain Adaptation and Few-Shot Learning50 references4 citations

TL;DR

This paper proposes an object-aware long-short-range spatial alignment framework for few-shot fine-grained image classification, integrating a foreground object enhancement (FOE) module, a long-range semantic correspondence (LSC) module for global feature matching, and a short-range spatial manipulation (SSM) module for local refinement. The method achieves state-of-the-art performance on four benchmark datasets under both 1-shot and 5-shot settings, demonstrating superior alignment of discriminative features across pose and viewpoint variations.

ABSTRACT

The goal of few-shot fine-grained image classification is to recognize rarely seen fine-grained objects in the query set, given only a few samples of this class in the support set. Previous works focus on learning discriminative image features from a limited number of training samples for distinguishing various fine-grained classes, but ignore one important fact that spatial alignment of the discriminative semantic features between the query image with arbitrary changes and the support image, is also critical for computing the semantic similarity between each support-query pair. In this work, we propose an object-aware long-short-range spatial alignment approach, which is composed of a foreground object feature enhancement (FOE) module, a long-range semantic correspondence (LSC) module and a short-range spatial manipulation (SSM) module. The FOE is developed to weaken background disturbance and encourage higher foreground object response. To address the problem of long-range object feature misalignment between support-query image pairs, the LSC is proposed to learn the transferable long-range semantic correspondence by a designed feature similarity metric. Further, the SSM module is developed to refine the transformed support feature after the long-range step to align short-range misaligned features (or local details) with the query features. Extensive experiments have been conducted on four benchmark datasets, and the results show superior performance over most state-of-the-art methods under both 1-shot and 5-shot classification scenarios.

Motivation & Objective

• To address the challenge of spatial misalignment in few-shot fine-grained image classification due to pose, scale, and viewpoint variations between support and query images.
• To improve feature matching by explicitly aligning discriminative object features across support and query images, beyond just learning discriminative features.
• To develop a multi-stage spatial alignment mechanism that combines long-range semantic correspondence with short-range refinement for improved fine-grained recognition.
• To enable effective few-shot adaptation in cross-domain fine-grained classification, such as recognizing rare bird species from generic bird datasets.

Proposed method

• The foreground object enhancement (FOE) module suppresses background interference and enhances responses to discriminative object parts in support and query images.
• The long-range semantic correspondence (LSC) module learns a transferable feature transformation using a learned similarity metric to align global semantic features between support and query images.
• The LSC module computes a semantic correlation matrix between support and query features to identify semantically matched regions across long-range spatial distances.
• The short-range spatial manipulation (SSM) module further refines the transformed support features by aligning local, fine-grained details with query features after long-range transformation.
• The three modules—FOE, LSC, and SSM—are stacked sequentially to progressively improve spatial alignment and feature similarity.
• The framework is trained end-to-end with a meta-learning paradigm, optimizing for few-shot classification accuracy on support-query pairs.

Experimental results

Research questions

• RQ1 Can explicit spatial alignment of discriminative features improve few-shot fine-grained classification beyond feature learning alone?
• RQ2 How effective is a long-range semantic correspondence module in aligning features across large spatial displacements due to pose and viewpoint changes?
• RQ3 Can short-range refinement after long-range alignment further improve local feature matching in fine-grained recognition?
• RQ4 Does the proposed method generalize to cross-domain few-shot fine-grained classification, such as recognizing rare species from a generic bird dataset?
• RQ5 How do the individual components (FOE, LSC, SSM) contribute to the overall performance gain?

Key findings

• The proposed method achieves state-of-the-art performance on four benchmark datasets—CUB, Stanford Cars, FG-WILDBEAST, and Birds-100—under both 1-shot and 5-shot few-shot learning settings.
• The LSC module significantly improves accuracy after FOE, demonstrating the effectiveness of long-range semantic correspondence in global feature alignment.
• The SSM module further boosts performance by refining local misalignments, showing that short-range refinement complements long-range transformation.
• On the CUB dataset, the method achieves 88.7% top-1 accuracy in the 5-shot setting, outperforming prior state-of-the-art methods.
• In cross-domain few-shot fine-grained classification, the method successfully adapts a generic bird classifier to recognize rare North American bird species with minimal labeled examples.
• Visualization results confirm that the semantic correlation matrix in LSC successfully identifies semantically matched regions, and the diagonal concentration in correlation matrices improves after SSM, indicating better spatial alignment.