[Paper Review] Sparse Spatial Transformers for Few-Shot Learning

Haoxing Chen, Huaxiong Li+2 more|arXiv (Cornell University)|Sep 27, 2021

Domain Adaptation and Few-Shot Learning15 references6 citations

TL;DR

This paper proposes Sparse Spatial Transformers (SSFormers), a novel transformer-based architecture for few-shot learning that enhances feature representation by extracting dense local image patches, using a sparse spatial transformer layer to select task-relevant patches across query and support images, and employing a patch-matching module for similarity computation. The method achieves state-of-the-art performance on few-shot learning benchmarks by jointly modeling task-specific context and suppressing irrelevant features.

ABSTRACT

Learning from limited data is challenging because data scarcity leads to a poor generalization of the trained model. A classical global pooled representation will probably lose useful local information. Many few-shot learning methods have recently addressed this challenge using deep descriptors and learning a pixel-level metric. However, using deep descriptors as feature representations may lose image contextual information. Moreover, most of these methods independently address each class in the support set, which cannot sufficiently use discriminative information and task-specific embeddings. In this paper, we propose a novel transformer-based neural network architecture called sparse spatial transformers (SSFormers), which finds task-relevant features and suppresses task-irrelevant features. Particularly, we first divide each input image into several image patches of different sizes to obtain dense local features. These features retain contextual information while expressing local information. Then, a sparse spatial transformer layer is proposed to find spatial correspondence between the query image and the full support set to select task-relevant image patches and suppress task-irrelevant image patches. Finally, we propose using an image patch-matching module to calculate the distance between dense local representations, thus determining which category the query image belongs to in the support set. Extensive experiments on popular few-shot learning benchmarks demonstrate the superiority of our method over state-of-the-art methods. Our source code is available at \url{https://github.com/chenhaoxing/ssformers}.

Motivation & Objective

• Address the limitations of global features and deep descriptors in few-shot learning, which lose local or contextual information.
• Overcome the inefficacy of independent class processing in existing metric-learning methods by leveraging task-level context.
• Develop a mechanism to identify and emphasize task-relevant image patches across query and support images.
• Create a task-specific prototype using spatial correspondence between query and support images to improve feature discrimination.
• Design a patch-matching module that computes patch-level similarity more effectively than standard cosine similarity.

Proposed method

• Divide input images into multiple image patches of varying sizes to extract dense local features while preserving contextual information.
• Introduce a sparse spatial transformer layer (SSTL) that uses cross-attention to find spatial correspondence between query patches and the full support set, selecting task-relevant patches and suppressing irrelevant ones.
• Apply a mutual nearest neighbor function to identify query patches with strong correlation to support set features, enabling selective feature aggregation.
• Construct task-specific prototypes by aggregating the most relevant query patches aligned with the support set using the SSTL.
• Implement a patch-matching module (PMM) that computes similarity scores between each query patch and its nearest neighbor in each aligned support prototype, accumulating scores for final classification.
• Use a nonparametric approach to match patches across query and support images, avoiding reliance on parametric similarity metrics.

Experimental results

Research questions

• RQ1 Can a transformer-based mechanism effectively identify task-relevant image patches across query and support images in few-shot learning?
• RQ2 Does modeling spatial correspondence between query and full support set improve feature discrimination compared to per-class processing?
• RQ3 Can patch-level similarity computation outperform global or pixel-level metrics in few-shot recognition?
• RQ4 How does the proposed method perform under distribution shifts and adversarial attacks compared to existing attention-based models?
• RQ5 Can dense local representations with spatial attention achieve better generalization than global features or deep descriptors?

Key findings

• SSFormers achieve state-of-the-art performance on mini-ImageNet, outperforming existing methods including CANs, FEAT, and RENet.
• The model maintains high robustness under various data corruptions, with only a 6.7% drop in accuracy under GaussianBlur, compared to a 40.0% drop for Rethink-D.
• The patch-matching module yields superior classification accuracy compared to direct cosine similarity, demonstrating its effectiveness in patch-level matching.
• t-SNE visualization confirms that SSFormers generate more discriminative and well-separated feature embeddings than Prototypical Net, even for unseen classes.
• Qualitative results show that the sparse spatial transformer successfully highlights relevant image regions and suppresses background or irrelevant features in both query and support images.
• The method exhibits low time complexity, with inference speed comparable to CANs, despite additional patch processing, due to efficient sparse attention.