[Paper Review] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Eleni Triantafillou, Tyler Zhu+9 more|arXiv (Cornell University)|Mar 7, 2019

Domain Adaptation and Few-Shot Learning30 references282 citations

TL;DR

Introduces Meta-Dataset, a large-scale, diverse few-shot learning benchmark across 10 datasets with varied tasks, evaluation protocols, and a novel meta-learner (Proto-MAML) to better generalize to new datasets.

ABSTRACT

Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

Motivation & Objective

• Motivate the need for a more realistic and diverse few-shot learning benchmark beyond Omniglot and mini-ImageNet.
• Provide a large-scale, multi-dataset environment to evaluate meta-learners on heterogeneous data and varying episode structures.
• Evaluate popular meta-learning models and baselines, and analyze how training data source, pre-training, and meta-learning affect generalization across datasets.
• Propose a new meta-learner (Proto-MAML) that combines prototypical networks with MAML-style adaptation to improve performance on Meta-Dataset.

Proposed method

• Define a new few-shot learning benchmark by aggregating 10 diverse datasets (ImageNet, Omniglot, Aircraft, CUB, Describable Textures, Quick Draw, Fungi, VGG Flowers, Traffic Signs, MSCOCO).
• Design a hierarchical, dataset-aware episode sampling procedure that varies ways (number of classes) and shots, and incorporates class structure (ImageNet/Omniglot hierarchies).
• Compare non-episodic baselines (k-NN, Finetune, cosine classifier variants) with episodic meta-learners (Matching Networks, Relation Networks, Prototypical Networks, MAML, Proto-MAML).
• Introduce Proto-MAML, a meta-learner initializing an episode-specific linear classifier from Prototypical Network prototypes and then allowing gradient-based adaptation of embeddings.
• Explore pre-training (ImageNet vs. all datasets), meta-training effects, and inference-only baselines to quantify benefits of meta-learning in heterogeneous data.
• Provide a reproducible evaluation protocol and open-source code for Meta-Dataset.

Experimental results

Research questions

• RQ1 Does a multi-dataset, heterogeneous episode design improve the ability of meta-learners to generalize to unseen datasets and tasks?
• RQ2 How do different training sources (ImageNet-only vs all datasets) and pre-training affect cross-dataset generalization?
• RQ3 Is meta-learning consistently beneficial across diverse datasets, or does its advantage depend on the data source and task configuration?
• RQ4 Can a hybrid meta-learner like Proto-MAML outperform standard meta-learners on a large, diverse benchmark?
• RQ5 What are the limits of current meta-learning approaches when faced with realistic class imbalance and variable shot/way settings?

Key findings

• Meta-Dataset provides a challenging, diverse benchmark showing that performance varies significantly across datasets and task configurations.
• Pre-training on ImageNet generally helps natural-image datasets but may hurt performance on more distant datasets (e.g., Omniglot, Quick Draw).
• Training on all datasets can improve some datasets (e.g., Omniglot, Quick Draw, Aircraft) but does not universally boost performance across all targets.
• Meta-learning benefits are not uniform when evaluating inference-only versus fully meta-trained baselines, especially under heterogeneous data; naive cross-dataset training does not automatically yield better generalization.
• Proto-MAML often yields the best average rank across datasets, with Prototypical Networks performing strongly and Finetune as a competitive baseline.
• The value of meta-learning depends on data heterogeneity, initialization, and the chosen episode-generation strategy, highlighting directions for future research.