[Paper Review] Adaptive Dimension Reduction and Variational Inference for Transductive Few-Shot Classification

Yuqing Hu, Stéphane Pateux+1 more|arXiv (Cornell University)|Sep 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes BAVARDAGE, a novel transductive few-shot classification method combining variational Bayesian inference with adaptive dimension reduction via probabilistic linear discriminant analysis (PLDA). It improves accuracy by up to 6% on unbalanced Few-Shot benchmarks and achieves state-of-the-art performance across multiple datasets, including new results in unbalanced settings.

ABSTRACT

Transductive Few-Shot learning has gained increased attention nowadays considering the cost of data annotations along with the increased accuracy provided by unlabelled samples in the domain of few shot. Especially in Few-Shot Classification (FSC), recent works explore the feature distributions aiming at maximizing likelihoods or posteriors with respect to the unknown parameters. Following this vein, and considering the parallel between FSC and clustering, we seek for better taking into account the uncertainty in estimation due to lack of data, as well as better statistical properties of the clusters associated with each class. Therefore in this paper we propose a new clustering method based on Variational Bayesian inference, further improved by Adaptive Dimension Reduction based on Probabilistic Linear Discriminant Analysis. Our proposed method significantly improves accuracy in the realistic unbalanced transductive setting on various Few-Shot benchmarks when applied to features used in previous studies, with a gain of up to $6\%$ in accuracy. In addition, when applied to balanced setting, we obtain very competitive results without making use of the class-balance artefact which is disputable for practical use cases. We also provide the performance of our method on a high performing pretrained backbone, with the reported results further surpassing the current state-of-the-art accuracy, suggesting the genericity of the proposed method.

Motivation & Objective

• Address the challenge of high uncertainty and poor statistical properties in few-shot classification due to limited labeled data.
• Improve performance in realistic, unbalanced transductive few-shot learning settings where class distribution among unlabelled samples varies significantly.
• Develop a method that generalizes across different backbones and datasets without relying on artificial class-balance assumptions.
• Enhance clustering robustness by integrating probabilistic modeling with dimensionality reduction to better capture class-specific distributions.

Proposed method

• Apply probabilistic linear discriminant analysis (PLDA) to perform adaptive dimension reduction, reducing feature space while preserving class-discriminative information.
• Use variational Bayesian (VB) inference to model cluster assignments and uncertainty, enabling flexible posterior estimation over class memberships.
• Integrate VB inference with PLDA to jointly optimize cluster structure and low-dimensional representation, improving estimation stability.
• Initialize clustering via Soft-KMEANS to provide robust starting points before iterative refinement using VB-PLDA.
• Apply the method end-to-end on features extracted from pretrained backbones (e.g., WRN, ResNet-12), enabling transferability across models.
• Use a 95% confidence interval to report accuracy with uncertainty quantification, enhancing reliability of results.

Experimental results

Research questions

• RQ1 Can variational Bayesian inference with adaptive dimension reduction improve few-shot classification accuracy in unbalanced transductive settings?
• RQ2 How does the proposed method perform compared to standard clustering baselines like Soft-KMEANS on both balanced and unbalanced Few-Shot benchmarks?
• RQ3 Does the method generalize across different backbone architectures and datasets without relying on artificial class-balance assumptions?
• RQ4 To what extent does the integration of PLDA and VB inference reduce uncertainty and improve statistical properties of learned clusters?

Key findings

• BAVARDAGE achieves a 6% accuracy gain on unbalanced transductive Few-Shot benchmarks compared to Soft-KMEANS, demonstrating significant improvement in realistic settings.
• On the CUB dataset, BAVARDAGE reaches 93.50% accuracy in the 5-shot unbalanced setting with RN12*, surpassing previous state-of-the-art results.
• The method achieves competitive performance in balanced settings without relying on class-balance artifacts, indicating robustness and generalization.
• On FC100 and CIFAR-FS, BAVARDAGE reports new state-of-the-art results in the unbalanced setting, which were previously untested in prior work.
• The method maintains high performance across multiple backbones (WRN, ResNet-18, ResNet-12), showing strong genericity and transferability.
• Average inference time is 1.72 seconds per 10,000 Few-Shot tasks on a single GPU, indicating practical efficiency.