[Paper Review] Learning Unsupervised Learning Rules

Luke Metz, Niru Maheswaranathan+2 more|arXiv (Cornell University)|Mar 31, 2018

Domain Adaptation and Few-Shot LearningComputer Science40 references40 citations

TL;DR

This paper proposes meta-learning an unsupervised learning rule—specifically, a biologically plausible, neuron-local weight update rule—that directly optimizes representations for downstream semi-supervised classification performance. The method generalizes across architectures, data modality (images to text), and input permutations, often outperforming standard unsupervised pretraining methods.

ABSTRACT

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this goal is approached by minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target a later desired task by meta-learning an unsupervised learning rule, which leads to representations useful for that task. Here, our desired task (meta-objective) is the performance of the representation on semi-supervised classification, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations that perform well under this meta-objective. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to novel neural network architectures. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.

Motivation & Objective

• To directly optimize unsupervised representations for downstream semi-supervised classification, rather than relying on surrogate objectives.
• To develop an unsupervised learning rule that generalizes across diverse neural network architectures, including varying widths, depths, and nonlinearities.
• To ensure biological plausibility by constraining the update rule to be neuron-local, enabling generalization to novel architectures.
• To evaluate generalization beyond image data, including to text tasks and permuted input dimensions.

Proposed method

• Meta-learn an unsupervised weight update rule using a meta-objective defined by downstream semi-supervised classification accuracy on the learned representations.
• Formulate the update rule as a differentiable, neuron-local function that depends only on local pre- and post-synaptic activity, mimicking biological learning rules.
• Train the update rule via gradient descent on a distribution of tasks, where each task involves pretraining a network with the rule and evaluating it on a downstream semi-supervised classification task.
• Use a bilevel optimization framework: inner loop trains the network with the meta-learned rule, outer loop updates the rule parameters based on downstream performance.
• Apply the meta-learned rule to train networks with varying widths, depths, and activation functions, assessing zero-shot generalization.
• Test generalization to data with randomly permuted input dimensions and to a text classification task using a transformer-based encoder.

Experimental results

Research questions

• RQ1 Can a meta-learned unsupervised update rule outperform standard unsupervised pretraining methods in downstream semi-supervised classification?
• RQ2 Does the meta-learned rule generalize to neural networks with different widths, depths, and nonlinearities?
• RQ3 Can the rule generalize to data with permuted input dimensions, indicating robustness to input reordering?
• RQ4 Does the rule transfer to a different modality, such as text classification, despite being trained on image data?
• RQ5 Is the learned rule biologically plausible and effective in a neuron-local, non-global update scheme?

Key findings

• The meta-learned unsupervised update rule produces representations that achieve competitive or superior performance on downstream semi-supervised classification tasks compared to standard unsupervised pretraining baselines.
• The rule generalizes effectively to neural networks with different widths, depths, and nonlinearities, demonstrating strong zero-shot transferability across architectures.
• The rule maintains strong performance when input dimensions are randomly permuted, indicating robustness to input reordering and generalization beyond specific data structures.
• The method generalizes from image data to a text classification task, showing transferability across modalities despite no fine-tuning on text.
• The biologically motivated, neuron-local update rule achieves performance comparable to or better than standard unsupervised methods, even without access to global gradients or labels during pretraining.