[Paper Review] 'Less Than One'-Shot Learning: Learning N Classes From M<N Samples

Ilia Sucholutsky, Matthias Schonlau|arXiv (Cornell University)|Sep 17, 2020

Domain Adaptation and Few-Shot Learning28 references27 citations

TL;DR

This paper introduces 'less than one'-shot learning (LO-shot), where N classes are learned from M < N soft-labeled prototypes. By using a distance-weighted k-Nearest Neighbors classifier with soft labels, the authors demonstrate that decision boundaries can separate any finite number of classes with as few as two prototypes, achieving sample efficiency far beyond hard-label methods and reducing minimal prototype requirements from O(N²) to O(1).

ABSTRACT

Deep neural networks require large training sets but suffer from high computational cost and long training times. Training on much smaller training sets while maintaining nearly the same accuracy would be very beneficial. In the few-shot learning setting, a model must learn a new class given only a small number of samples from that class. One-shot learning is an extreme form of few-shot learning where the model must learn a new class from a single example. We propose the `less than one'-shot learning task where models must learn $N$ new classes given only $M<N$ examples and we show that this is achievable with the help of soft labels. We use a soft-label generalization of the k-Nearest Neighbors classifier to explore the intricate decision landscapes that can be created in the `less than one'-shot learning setting. We analyze these decision landscapes to derive theoretical lower bounds for separating $N$ classes using $M<N$ soft-label samples and investigate the robustness of the resulting systems.

Motivation & Objective

• To establish the theoretical feasibility of learning N classes from fewer than N training samples.
• To investigate how soft labels enable effective decision landscape formation in extreme low-shot regimes.
• To analyze the robustness and stability of decision boundaries created under LO-shot constraints.
• To demonstrate that soft-label prototypes can drastically reduce the number of required training samples compared to hard-label methods.
• To provide a theoretical foundation for dataset condensation and prototype generation in low-data regimes.

Proposed method

• Proposes a soft-label generalization of the k-Nearest Neighbors classifier (SLaPkNN) that uses distance-weighted voting with probabilistic label distributions.
• Employs soft labels to encode partial class membership, allowing each prototype to contribute to multiple class boundaries.
• Derives theoretical lower bounds for separating N classes using M < N soft-label prototypes through geometric and probabilistic analysis.
• Analyzes decision landscape robustness by perturbing prototype positions and soft labels, measuring risk gradients and intra-class risk behavior.
• Uses a case study on concentric circles to compare hard-label and soft-label prototype generation, showing superior performance with soft labels.
• Applies inverse distance weighting and soft label fusion to compute class probabilities, enabling multi-class separation from minimal prototypes.

Experimental results

Research questions

• RQ1 Can N classes be separated using M < N soft-labeled prototypes in a decision space?
• RQ2 What are the theoretical lower bounds on the number of soft-label prototypes required to separate N classes?
• RQ3 How does the robustness of LO-shot decision landscapes vary under perturbations of prototype positions and labels?
• RQ4 Can soft-label prototypes reduce the minimal number of required prototypes from O(N²) to O(1) in structured data?
• RQ5 How do soft labels improve sample efficiency compared to hard-label prototypes in prototype-based classification?

Key findings

• A single SLaPkNN classifier fitted on just two soft-label prototypes can partition the decision space into any finite number of classes.
• Theoretical analysis shows that soft-label prototypes can achieve perfect separation of N classes with M < N samples, defying classical intuition.
• Robustness analysis reveals that carefully tuned prototype configurations can stabilize decision landscapes and minimize intra-class risk fluctuations.
• In the concentric circles case study, soft-label prototypes reduced the minimal number of required prototypes from O(N²) to O(1), with five prototypes separating six classes.
• The method achieves high sample efficiency: five soft-label prototypes trained on MNIST can yield over 90% accuracy, demonstrating feasibility of LO-shot learning.
• Soft-label SLaPkNN outperforms hard-label kNN in geometric fidelity, producing smooth, ellipse-like decision boundaries that match underlying data structure.