[Paper Review] Few-Shot Unsupervised Continual Learning through Meta-Examples

Alessia Bertugli, Stefano Vincenzi+2 more|arXiv (Cornell University)|Sep 17, 2020

Domain Adaptation and Few-Shot Learning48 references4 citations

TL;DR

This paper proposes FUSION, a novel few-shot unsupervised continual learning setting with unbalanced, clustering-derived tasks, and introduces MEML, a meta-learning method that uses a self-attention mechanism to form a single 'meta-example' for efficient inner-loop updates. The approach achieves competitive performance on Omniglot and Mini-ImageNet, even surpassing supervised baselines in some cases, and demonstrates that small, unbalanced clusters and feature variability enhance generalization over data balance.

ABSTRACT

In real-world applications, data do not reflect the ones commonly used for neural networks training, since they are usually few, unlabeled and can be available as a stream. Hence many existing deep learning solutions suffer from a limited range of applications, in particular in the case of online streaming data that evolve over time. To narrow this gap, in this work we introduce a novel and complex setting involving unsupervised meta-continual learning with unbalanced tasks. These tasks are built through a clustering procedure applied to a fitted embedding space. We exploit a meta-learning scheme that simultaneously alleviates catastrophic forgetting and favors the generalization to new tasks. Moreover, to encourage feature reuse during the meta-optimization, we exploit a single inner loop taking advantage of an aggregated representation achieved through the use of a self-attention mechanism. Experimental results on few-shot learning benchmarks show competitive performance even compared to the supervised case. Additionally, we empirically observe that in an unsupervised scenario, the small tasks and the variability in the clusters pooling play a crucial role in the generalization capability of the network. Further, on complex datasets, the exploitation of more clusters than the true number of classes leads to higher results, even compared to the ones obtained with full supervision, suggesting that a predefined partitioning into classes can miss relevant structural information.

Motivation & Objective

• To address the gap in real-world continual learning where data are few, unlabeled, and unbalanced, by proposing a new realistic setting.
• To develop a meta-learning framework that mitigates catastrophic forgetting and generalizes well to new, unbalanced tasks in unsupervised continual learning.
• To investigate the impact of unbalanced clusters and feature variability on model generalization in few-shot unsupervised learning.
• To demonstrate that predefined class partitions may miss structural information, and that using more clusters than true classes improves performance.

Proposed method

• The method uses an embedding network (Deep Cluster or ACAI) to learn a latent space from unlabeled data before applying k-means clustering to form unbalanced tasks.
• Tasks are constructed from clusters with varying sizes, preserving data distribution and avoiding artificial balancing or cluster regularization.
• MEML employs a double-loop meta-learning scheme: a single inner loop update using a self-attention mechanism to aggregate the most representative features into a 'meta-example'.
• The outer loop performs full model optimization using a combination of task-specific data and randomly sampled data from the overall training trajectory to prevent forgetting.
• Self-attention coefficients are computed via a learnable network to identify the most informative samples within each cluster, enabling efficient and focused feature reuse.
• The model is trained with Adam optimizer, using a meta-batch size of 1, and tested on Omniglot, Mini-ImageNet, SlimageNet64, and Cifar100 for out-of-distribution evaluation.

Experimental results

Research questions

• RQ1 How does the performance of meta-learning models change when tasks are unbalanced and derived from unsupervised clustering rather than balanced, predefined classes?
• RQ2 What is the impact of using small or unbalanced clusters on generalization and catastrophic forgetting in continual learning?
• RQ3 Can a self-attention mechanism that forms a single meta-example improve efficiency and performance in few-shot unsupervised continual learning?
• RQ4 Does using more clusters than the true number of classes lead to better performance than supervised baselines or fixed class partitioning?
• RQ5 How does the model generalize to out-of-distribution datasets like Cifar100 when trained on Omniglot or Mini-ImageNet?

Key findings

• On Omniglot, the MEML model achieves 91.1% accuracy in the 5-way 1-shot setting, outperforming supervised baselines and showing strong generalization in unsupervised continual learning.
• On Mini-ImageNet, the model reaches 57.4% accuracy in the 5-way 1-shot setting, demonstrating competitive performance despite the absence of labels during training.
• The ablation study confirms that updating on the self-attention-derived meta-example is superior to random sampling or multiple updates on the same task, reducing training time and memory usage.
• Empirically, unbalanced clusters and feature variability are more critical for generalization than data balance, suggesting that small clusters contribute meaningfully to learning.
• Using more clusters than the true number of classes leads to higher performance than full supervision on complex datasets, indicating that predefined class boundaries may miss underlying structural information.
• On Cifar100, the model fine-tuned with FiLM layers on Omniglot data achieves 50.0% accuracy in the 2-way 1-shot setting, showing transferability to out-of-distribution tasks.