[Paper Review] Finding Significant Features for Few-Shot Learning using Dimensionality Reduction

Mauricio Méndez-Ruiz, Iván García+2 more|arXiv (Cornell University)|Jul 6, 2021

Domain Adaptation and Few-Shot Learning25 references1 citations

TL;DR

This paper proposes a novel ICNN module that uses dimensionality reduction and an inter/intra-class nearest-neighbor score to identify task-significant features for few-shot learning. By selecting the most discriminative features before metric learning, the method improves accuracy by ~2% over baselines on miniImageNet, especially in 5-way 1-shot settings.

ABSTRACT

Few-shot learning is a relatively new technique that specializes in problems where we have little amounts of data. The goal of these methods is to classify categories that have not been seen before with just a handful of samples. Recent approaches, such as metric learning, adopt the meta-learning strategy in which we have episodic tasks conformed by support (training) data and query (test) data. Metric learning methods have demonstrated that simple models can achieve good performance by learning a similarity function to compare the support and the query data. However, the feature space learned by a given metric learning approach may not exploit the information given by a specific few-shot task. In this work, we explore the use of dimension reduction techniques as a way to find task-significant features helping to make better predictions. We measure the performance of the reduced features by assigning a score based on the intra-class and inter-class distance, and selecting a feature reduction method in which instances of different classes are far away and instances of the same class are close. This module helps to improve the accuracy performance by allowing the similarity function, given by the metric learning method, to have more discriminative features for the classification. Our method outperforms the metric learning baselines in the miniImageNet dataset by around 2% in accuracy performance.

Motivation & Objective

• Address the challenge of limited data in few-shot learning by identifying features most relevant to a given task.
• Improve metric learning performance by focusing on discriminative features rather than full-dimensional embeddings.
• Overcome the limitation of standard metric learning methods that may not exploit task-specific feature relevance.
• Develop a general-purpose module compatible with various metric learning frameworks to enhance feature representation for few-shot classification.

Proposed method

• Apply multiple dimensionality reduction techniques (PCA, Isomap, UMAP) to support set embeddings from a feature extractor.
• Compute an ICNN score based on inter-class and intra-class nearest-neighbor distances to evaluate feature reduction quality.
• Select the best-performing dimensionality reduction method using the ICNN score for each episodic task.
• Apply the selected reduction method to both support and query embeddings before metric learning inference.
• Integrate the ICNN module as a plug-in to existing metric learning models like Prototypical Networks.
• Use episodic training with random task sampling from miniImageNet to optimize the feature selection process.

Experimental results

Research questions

• RQ1 Can dimensionality reduction techniques improve few-shot classification by identifying task-significant features?
• RQ2 How does an inter- and intra-class distance-based scoring mechanism compare to standard feature representations in few-shot learning?
• RQ3 Does selecting task-specific features through ICNN improve generalization and robustness in low-shot scenarios?
• RQ4 How does the proposed method compare to state-of-the-art models in terms of accuracy and stability across different few-shot settings?

Key findings

• The ICNN module improved Prototypical Networks by approximately 2% in accuracy on the 5-way 1-shot setting of miniImageNet.
• The method achieved a 63.81% ± 0.71 test accuracy on 5-way 1-shot and 78.14% ± 0.50 on 5-way 5-shot using ResNet-12 as the backbone.
• Using both support and query data during dimensionality reduction led to significantly better ICNN scores than using support data alone.
• The ICNN module improved performance across all quartiles in accuracy distributions, with notable gains in the first and fourth quartiles, indicating enhanced robustness.
• The method outperformed standard metric learning baselines (e.g., Matching Networks, Relation Networks) and state-of-the-art models like K-tuplet Nets and Principal Characteristic Nets.
• The best-performing feature reduction techniques were PCA, Isomap, and UMAP, while more complex models degraded performance due to poor prototype estimation.