[Paper Review] Meta-Learning Update Rules for Unsupervised Representation Learning

Luke Metz, Niru Maheswaranathan+2 more|arXiv (Cornell University)|Mar 31, 2018

Domain Adaptation and Few-Shot Learning64 references69 citations

TL;DR

The paper meta-learns an unsupervised, neuron-local update rule that trains a base model to produce representations useful for semi-supervised classification, and demonstrates strong generalization across architectures, datasets, and modalities.

ABSTRACT

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.

Motivation & Objective

• Motivate unsupervised representation learning as a target for meta-learning to improve downstream tasks.
• Directly optimize representations for semi-supervised classification without labels during training.
• Develop a neuron-local unsupervised update rule that generalizes across architectures and data domains.
• Show that the meta-learned update rule can transfer to different widths, depths, nonlinearities, and data modalities.

Proposed method

• Formulate an MLP base model with layerwise, neuron-local update networks parameterized by meta-parameters θ.
• Define an unsupervised update rule that uses per-neuron hidden states and local pre-/post-synaptic activity for updates.
• Backpropagate a meta-objective through time to optimize θ via truncated backpropagation through time.
• Meta-objective minimizes a cosine distance between held-out labeled targets and linear regression predictions from learned features.
• Train on diverse datasets and base-model architectures to promote generalization across domains and modalities.
• Permute input dimensions during meta-training to enforce permutation-invariance of the learning rule.

Experimental results

Research questions

• RQ1 Can a learned unsupervised update rule produced via meta-learning generate representations useful for few-shot classification?
• RQ2 Does the meta-learned rule generalize to different network widths, depths, activation functions, and data modalities?
• RQ3 To what extent does varying meta-training data and architectures promote cross-domain generalization?
• RQ4 Is a neuron-local update rule compatible with biologically plausible learning while achieving competitive unsupervised representation quality?
• RQ5 How does meta-learning the update rule compare to traditional hand-designed unsupervised objectives in terms of transferability?

Key findings

• The meta-learned unsupervised update rule yields representations that outperform some existing unsupervised methods on held-out tasks.
• The learned rule generalizes to base models with unseen widths, depths, and nonlinearities.
• The rule transfers across data modalities, including from image datasets to a text task, though with varying degrees of success.
• Performance remains robust under input dimension permutations, demonstrating permutation-invariant learning behavior.
• Meta-training with diverse datasets and architectures improves generalization across domains.
• Visual analyses show the first-layer filters evolve from coarse/noise to more structured, dataset-appropriate features over meta-training.