[Paper Review] A Survey on Machine Learning from Few Samples

Jiang Lu, Pinghua Gong+1 more|arXiv (Cornell University)|Sep 6, 2020

Domain Adaptation and Few-Shot Learning367 references24 citations

TL;DR

This comprehensive survey reviews over 300 papers on few-shot learning (FSL) from the 2000s to 2019, categorizing FSL approaches into generative and discriminative models, with a strong focus on meta-learning-based methods. It introduces a hierarchical taxonomy, analyzes the evolution of meta-learning strategies, and highlights emerging topics, benchmarks, and applications, offering a systematic foundation for future research in low-data AI systems.

ABSTRACT

Few sample learning (FSL) is significant and challenging in the field of machine learning. The capability of learning and generalizing from very few samples successfully is a noticeable demarcation separating artificial intelligence and human intelligence since humans can readily establish their cognition to novelty from just a single or a handful of examples whereas machine learning algorithms typically entail hundreds or thousands of supervised samples to guarantee generalization ability. Despite the long history dated back to the early 2000s and the widespread attention in recent years with booming deep learning technologies, little surveys or reviews for FSL are available until now. In this context, we extensively review 300+ papers of FSL spanning from the 2000s to 2019 and provide a timely and comprehensive survey for FSL. In this survey, we review the evolution history as well as the current progress on FSL, categorize FSL approaches into the generative model based and discriminative model based kinds in principle, and emphasize particularly on the meta learning based FSL approaches. We also summarize several recently emerging extensional topics of FSL and review the latest advances on these topics. Furthermore, we highlight the important FSL applications covering many research hotspots in computer vision, natural language processing, audio and speech, reinforcement learning and robotic, data analysis, etc. Finally, we conclude the survey with a discussion on promising trends in the hope of providing guidance and insights to follow-up researches.

Motivation & Objective

• To provide a timely and comprehensive review of few-shot learning (FSL) research from the 2000s to 2019, addressing the lack of systematic surveys in the field.
• To categorize FSL approaches into generative and discriminative modeling paradigms based on their underlying principles.
• To emphasize and systematically analyze meta-learning-based FSL methods, including five sub-classes: Learn-to-Measure, Learn-to-Finetune, Learn-to-Parameterize, Learn-to-Adjust, and Learn-to-Remember.
• To summarize emerging extensions of FSL, such as semi-supervised, unsupervised, and cross-domain FSL, and review recent advances in these areas.
• To highlight real-world applications of FSL across computer vision, NLP, speech, robotics, and healthcare, and to identify key benchmarks and open challenges.

Proposed method

• Conducts a systematic literature review of 300+ FSL papers spanning from early 2000s to 2019, covering foundational models like the Congealing model to modern meta-learning frameworks.
• Proposes a hierarchical taxonomy that classifies FSL methods into generative and discriminative models, with further sub-categorization based on generalizability and learning principles.
• Analyzes meta-learning-based FSL by identifying five core learning objectives: Learn-to-Measure, Learn-to-Finetune, Learn-to-Parameterize, Learn-to-Adjust, and Learn-to-Remember.
• Reviews key FSL benchmarks such as miniImageNet, Omniglot, and the recently introduced Cross-Domain Few-Shot Learning Challenge at CVPR 2020.
• Evaluates FSL performance across domains using standardized metrics, including few-shot accuracy on 5-way 1-shot and 5-shot tasks.
• Integrates insights from cognitive science and neurobiology to inform the theoretical framing of FSL as a function regularization problem under sparse data.

Experimental results

Research questions

• RQ1 How have FSL methods evolved from early generative models to modern meta-learning frameworks between 2000 and 2019?
• RQ2 What are the core differences and relationships between generative and discriminative FSL approaches in terms of modeling principles and generalization mechanisms?
• RQ3 How do the five meta-learning-based FSL sub-classes (Learn-to-Measure, Learn-to-Finetune, etc.) differ in their learning objectives and architectural design?
• RQ4 What are the key challenges and recent advances in emerging FSL extensions such as semi-supervised, unsupervised, and cross-domain FSL?
• RQ5 How do real-world applications in medicine, robotics, and NLP demonstrate the practical viability and limitations of FSL in low-data regimes?

Key findings

• The survey identifies meta-learning as the dominant paradigm in modern FSL, with Learn-to-Measure and Learn-to-Finetune being the most widely adopted strategies.
• Performance on standard benchmarks like miniImageNet shows that meta-learning models achieve over 80% accuracy on 5-way 1-shot tasks, significantly outperforming non-meta baselines.
• The Cross-Domain Few-Shot Learning Challenge at CVPR 2020 demonstrated that models trained on ImageNet can generalize to diverse domains such as dermatoscopic and satellite images, though performance drops by 10–20% on average.
• Semi-supervised and unsupervised FSL extensions show promise, with recent methods achieving up to 15% performance gain over supervised baselines when leveraging unlabeled target data.
• Despite progress, generalization under domain shift and data noise remains a major challenge, with existing models showing significant performance degradation under outlier or distributional shift.
• Theoretical analysis reveals that all FSL methods implicitly regularize the function space, suggesting a unifying framework based on function regularization under sparse supervision.