[Paper Review] Function Contrastive Learning of Transferable Meta-Representations

Muhammad Waleed Gondal, Shruti Joshi+4 more|arXiv (Cornell University)|Oct 14, 2020

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes Function Contrastive Representation Learning (FCRL), a decoupled encoder-decoder framework that pretrains a meta-representation encoder using a contrastive objective over unordered, partial observations of functions. By contrasting context sets from the same vs. different functions, FCRL learns robust, transferable meta-representations that outperform end-to-end trained baselines in downstream few-shot prediction and noise-robustness, especially in vision and control tasks.

ABSTRACT

Meta-learning algorithms adapt quickly to new tasks that are drawn from the same task distribution as the training tasks. The mechanism leading to fast adaptation is the conditioning of a downstream predictive model on the inferred representation of the task's underlying data generative process, or \emph{function}. This \emph{meta-representation}, which is computed from a few observed examples of the underlying function, is learned jointly with the predictive model. In this work, we study the implications of this joint training on the transferability of the meta-representations. Our goal is to learn meta-representations that are robust to noise in the data and facilitate solving a wide range of downstream tasks that share the same underlying functions. To this end, we propose a decoupled encoder-decoder approach to supervised meta-learning, where the encoder is trained with a contrastive objective to find a good representation of the underlying function. In particular, our training scheme is driven by the self-supervision signal indicating whether two sets of examples stem from the same function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.

Motivation & Objective

• To improve the transferability and noise robustness of meta-representations in few-shot learning.
• To address the limitations of joint encoder-decoder training in meta-learning, which can lead to supervision collapse and overfitting to task-specific noise.
• To decouple meta-representation learning from downstream prediction tasks using self-supervision.
• To enable a single, shared meta-representation to generalize across diverse downstream tasks.
• To develop a representation that captures essential functional structure while ignoring irrelevant noise and background features.

Proposed method

• The method uses a contrastive learning objective to train an encoder to produce similar representations for context sets drawn from the same underlying function.
• Positive pairs are formed by sampling different subsets of the same function; negative pairs are formed from different functions.
• The contrastive loss is computed using a momentum encoder and a momentum queue to stabilize training.
• The pre-trained encoder is then used to produce meta-representations that condition separate, task-specific decoders during inference.
• The framework is applied to both synthetic and real-world datasets, including vision and reinforcement learning tasks.
• The approach avoids reconstruction-based objectives, focusing instead on functional similarity to improve generalization.

Experimental results

Research questions

• RQ1 Can a contrastive objective over context sets improve the transferability of meta-representations across diverse downstream tasks?
• RQ2 Does decoupling meta-representation learning from downstream prediction enhance robustness to data noise?
• RQ3 How does FCRL compare to end-to-end meta-learning baselines in few-shot regression and classification?
• RQ4 Can FCRL representations support data-efficient policy learning in visual control tasks?
• RQ5 Does the absence of reconstruction loss lead to better disentanglement of relevant functional features from background noise?

Key findings

• FCRL outperforms strong end-to-end meta-learning baselines in downstream few-shot prediction tasks, even when those baselines are trained with full supervision.
• The method achieves superior robustness to noise, particularly in preserving structural details like object shapes while ignoring static backgrounds.
• In a visual control task, the FCRL-based policy significantly outperforms the GQN-based policy, demonstrating improved data efficiency.
• The contrastive pretraining of the encoder leads to more disentangled and generalizable representations compared to joint training.
• The decoupled training scheme enables the same pre-trained encoder to be fine-tuned efficiently for multiple downstream tasks without retraining.
• The method shows strong performance on both synthetic functions and real-world datasets, including image generation and visual question answering.