[Paper Review] On the Importance of Firth Bias Reduction in Few-Shot Classification

Saba Ghaffari, Ehsan Saleh+2 more|arXiv (Cornell University)|Oct 6, 2021

Domain Adaptation and Few-Shot Learning86 references4 citations

TL;DR

This paper proposes Firth bias reduction as a simple yet highly effective regularization technique for few-shot image classification, reformulating it as KL-divergence penalization between uniform class distribution and model predictions. It demonstrates consistent 0.5–3% accuracy gains across diverse backbones, data settings, and classifier types—outperforming L2, label smoothing, and data augmentation, especially in low-shot and imbalanced regimes.

ABSTRACT

Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision. The performance in few-shot classification suffers from the bias in the estimation of classifier parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classifiers has been overlooked. In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classification. Theoretically, Firth bias reduction removes the $O(N^{-1})$ first order term from the small-sample bias of the Maximum Likelihood Estimator. Here we show that the general Firth bias reduction technique simplifies to encouraging uniform class assignment probabilities for multinomial logistic classification, and almost has the same effect in cosine classifiers. We derive an easy-to-implement optimization objective for Firth penalized multinomial logistic and cosine classifiers, which is equivalent to penalizing the cross-entropy loss with a KL-divergence between the uniform label distribution and the predictions. Then, we empirically evaluate that it is consistently effective across the board for few-shot image classification, regardless of (1) the feature representations from different backbones, (2) the number of samples per class, and (3) the number of classes. Finally, we show the robustness of Firth bias reduction, in the case of imbalanced data distribution. Our implementation is available at https://github.com/ehsansaleh/firth_bias_reduction

Motivation & Objective

• Address the underappreciated issue of parameter estimation bias in few-shot classifiers trained on very few samples.
• Investigate whether Firth bias reduction—known in statistics for reducing small-sample bias—can improve few-shot learning performance.
• Demonstrate that Firth penalization is effective regardless of feature backbone, class imbalance, number of classes, or classifier architecture.
• Show that Firth bias reduction cannot be replicated by standard regularization techniques like L2 or label smoothing.
• Establish Firth penalization as a robust, general-purpose solution for improving small-sample classifier training in few-shot learning.

Proposed method

• Reformulate Firth penalized maximum likelihood estimation (PMLE) as a KL-divergence penalty between uniform label distribution and model predictions, enabling easy integration into standard training pipelines.
• Derive a simplified optimization objective that adds a log-determinant penalty (log det(F)) to the cross-entropy loss, equivalent to KL-divergence minimization between uniform prior and output probabilities.
• Apply the Firth penalty to both multinomial logistic and cosine classifiers, showing its geometric and statistical equivalence in both cases.
• Implement the method as a differentiable, layer-agnostic loss term that can be used with any backbone and classifier head.
• Use empirical evaluation across multiple datasets (tiered-ImageNet, CIFAR-FS), backbones (ResNet, DenseNet, MobileNet), and few-shot settings (1–5 shots, 5–16 ways).
• Compare against strong baselines: L2 regularization, confidence penalty, unigram label smoothing, and data augmentation with artificial samples.

Experimental results

Research questions

• RQ1 Can Firth bias reduction consistently improve few-shot image classification accuracy across diverse backbone networks and data distributions?
• RQ2 Does Firth penalization outperform standard regularization techniques like L2 and label smoothing in small-sample regimes?
• RQ3 How effective is Firth bias reduction in the presence of class imbalance or when using artificial data augmentation?
• RQ4 Can Firth penalization be generalized beyond standard logistic models, such as to cosine classifiers used in meta-learning?
• RQ5 Is the improvement from Firth penalization statistically significant and robust across multiple few-shot benchmarks and hyperparameter settings?

Key findings

• Firth bias reduction consistently improves few-shot classification accuracy by 0.5% to 3% across all evaluated backbones, including ResNet, DenseNet, and MobileNet.
• The improvement is statistically significant in all tested settings, including balanced and imbalanced data distributions, and across 1–16 shot and 5–16 way classification tasks.
• Firth penalization outperforms L2 regularization, which shows mostly insignificant improvements in the same few-shot settings.
• Firth bias reduction also surpasses advanced label smoothing variants (confidence penalty and unigram smoothing), which cannot replicate its bias-reduction effect.
• Even when artificial data augmentation is used (e.g., 750 synthetic samples per class), Firth penalization still provides significant gains, indicating it addresses fundamental estimation bias beyond data scarcity.
• In cross-domain few-shot settings, Firth penalization remains effective where data augmentation fails due to domain shift, highlighting its robustness to distributional shift.