[Paper Review] Unsupervised Learning via Meta-Learning

Kyle Hsu, Sergey Levine+1 more|arXiv (Cornell University)|Oct 4, 2018

Domain Adaptation and Few-Shot Learning55 references128 citations

TL;DR

The paper proposes CACTUs, an unsupervised meta-learning framework that automatically constructs supervised-like tasks from unlabeled data via embeddings and clustering, enabling effective few-shot learning for downstream tasks without any labels.

ABSTRACT

A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disentanglement, prediction, and other metrics. Instead, we develop an unsupervised meta-learning method that explicitly optimizes for the ability to learn a variety of tasks from small amounts of data. To do so, we construct tasks from unlabeled data in an automatic way and run meta-learning over the constructed tasks. Surprisingly, we find that, when integrated with meta-learning, relatively simple task construction mechanisms, such as clustering embeddings, lead to good performance on a variety of downstream, human-specified tasks. Our experiments across four image datasets indicate that our unsupervised meta-learning approach acquires a learning algorithm without any labeled data that is applicable to a wide range of downstream classification tasks, improving upon the embedding learned by four prior unsupervised learning methods.

Motivation & Objective

• Enable efficient downstream learning from unlabeled data by optimizing a learning procedure rather than a proxy unsupervised objective.
• Automatically construct structured learning tasks from unlabeled data using embeddings and clustering.
• Demonstrate transfer to diverse human-designed downstream tasks across multiple image datasets.
• Show that the learned learning procedure can outperform embedding-only baselines and approach supervised meta-learning under certain conditions.

Proposed method

• Input unlabeled dataset and learn embeddings with an unsupervised embedding method E.
• Generate multiple partitions of the embedding space by running k-means with random scaling to create diverse task distributions.
• Construct M-way, K-shot classification tasks from selected clusters to form meta-training tasks without labels.
• Apply meta-learning algorithms (MAML and ProtoNets) to learn a procedure F that quickly adapts to new tasks.
• Evaluate the learned learning procedure on downstream human-designed tasks (character recognition, object classification, facial attribute discrimination) across multiple datasets.
• Compare CACTUs-based meta-learning to embedding-based baselines and to oracle supervised meta-learning.

Experimental results

Research questions

• RQ1 Does unsupervised meta-learning (CACTUs) yield a learning procedure that improves downstream task performance over embedding-based methods?
• RQ2 Is CACTUs effective across different embedding spaces and unsupervised learning methods?
• RQ3 Can the learned learning procedure transfer to a variety of downstream tasks and shot settings (1-shot to 50-shot)?
• RQ4 How does unsupervised CACTUs compare to supervised meta-learning with hand-designed task distributions (oracle)?

Key findings

• CACTUs-MAML consistently outperforms embedding-based baselines on downstream few-shot tasks across several datasets.
• CACTUs-ProtoNets also show improved performance, though ProtoNets may underperform in some settings when meta-training and testing shots are mismatched.
• CACTUs yields usable priors for a range of downstream tasks (character, object, and facial attribute tasks) without any labels during meta-training.
• The choice of embedding space affects performance, and better embedding methods correlate with stronger unsupervised meta-learning results.
• Non-random task construction (via clustering in embedding space) substantially outperforms random or pixel-space task construction, and over multiple partitions adds robustness.
• Performance gaps relative to an oracle supervised meta-learning setup depend on task difficulty and data overlap, with easier tasks showing smaller gaps.