[Paper Review] Regression Networks for Meta-Learning Few-Shot Classification

Arnout Devos, Matthias Grossglauser|arXiv (Cornell University)|May 31, 2019

Domain Adaptation and Few-Shot Learning29 references4 citations

TL;DR

This paper proposes Regression Networks for few-shot image classification, where a query point is classified by regressing its closest approximation within each class's embedded subspace and using the regression error as a distance metric. The method achieves state-of-the-art performance, especially with higher shot settings, by leveraging directional information in high-dimensional embeddings and robust class subspace modeling.

ABSTRACT

We propose regression networks for the problem of few-shot classification, where a classifier must generalize to new classes not seen in the training set, given only a small number of examples of each class. In high dimensional embedding spaces the direction of data generally contains richer information than magnitude. Next to this, state-of-the-art few-shot metric methods that compare distances with aggregated class representations, have shown superior performance. Combining these two insights, we propose to meta-learn classification of embedded points by regressing the closest approximation in every class subspace while using the regression error as a distance metric. Similarly to recent approaches for few-shot learning, regression networks reflect a simple inductive bias that is beneficial in this limited-data regime and they achieve excellent results, especially when more aggregate class representations can be formed with multiple shots.

Motivation & Objective

• Address the challenge of few-shot image classification, where models must generalize to novel classes with only a few labeled examples.
• Overcome limitations of existing metric-based methods that rely on simple centroid comparisons and may underutilize directional information in high-dimensional embeddings.
• Improve generalization under domain shift by learning more robust, structured representations through regression-based class subspace modeling.
• Develop a simple yet effective meta-learning framework that reflects a beneficial inductive bias in low-data regimes.
• Demonstrate superior performance across multiple few-shot benchmarks, particularly when more than one shot per class is available.

Proposed method

• Map input images to a high-dimensional embedding space using a neural network with learnable parameters φ.
• For each class, construct a K-dimensional subspace matrix Sₙ from the K support set embeddings fₚ(𝐱ₙᵢ).
• For each query point eᵢ, regress its best approximation within each class subspace Sₙ using a learned linear mapping.
• Compute the regression error d̃(eᵢ, Sₙ) as a measure of distance between the query and the class subspace.
• Classify the query point by selecting the class with the minimum regression error.
• Use a subspace orthogonalization regularizer during training to improve generalization and reduce redundancy in learned subspaces.

Experimental results

Research questions

• RQ1 Can regression-based approximation within class subspaces improve few-shot classification beyond standard metric learning with class centroids?
• RQ2 How does leveraging directional information in high-dimensional embeddings affect performance in low-shot settings?
• RQ3 Does modeling class representations as subspaces rather than single points lead to better generalization under domain shift?
• RQ4 How does the performance of regression networks scale with increasing shot count (e.g., 1-shot vs. 5-shot vs. 10-shot)?
• RQ5 How does the method compare to state-of-the-art metric learning baselines like ProtoNet and RelationNet under both in-domain and out-of-domain evaluation?

Key findings

• Regression Networks achieve state-of-the-art performance on mini-ImageNet for 5-way 1-shot, 5-shot, and 10-shot classification across multiple backbones (Conv-4, ResNet-10, ResNet-34).
• On 5-way 1-shot mini-ImageNet with a ResNet-10 backbone, RegressionNet achieves 74.03% accuracy, outperforming ProtoNet (73.77%) and RelationNet (69.97%).
• In the challenging 5-way 5-shot mini-ImageNet → CUB domain shift setting, RegressionNet achieves 62.71% accuracy, significantly outperforming ProtoNet (59.22%) and RelationNet (54.36%).
• RegressionNet maintains strong performance under domain shift, with 56.66% accuracy in the CUB → mini-ImageNet setting, outperforming all baselines.
• The method shows consistent improvement over baselines as shot count increases, indicating that the subspace modeling approach benefits from more support examples.
• The inclusion of the subspace orthogonalization regularizer improves generalization, particularly in domain shift scenarios, confirming its utility in low-data regimes.