[Paper Review] Unsupervised Transfer Learning with Self-Supervised Remedy

Jiabo Huang, Shaogang Gong|arXiv (Cornell University)|Jun 8, 2020

Domain Adaptation and Few-Shot Learning45 references4 citations

TL;DR

This paper proposes SUPREME, a self-supervised remedy for unsupervised transfer learning that enhances transfer clustering by combining prior knowledge from labeled source domains with self-supervised pairwise similarity learning on unlabelled target data. The method improves clustering performance across four datasets and achieves competitive results on zero-shot learning benchmarks without relying on textual or attribute-based supervision.

ABSTRACT

Generalising deep networks to novel domains without manual labels is challenging to deep learning. This problem is intrinsically difficult due to unpredictable changing nature of imagery data distributions in novel domains. Pre-learned knowledge does not transfer well without making strong assumptions about the learned and the novel domains. Different methods have been studied to address the underlying problem based on different assumptions, e.g. from domain adaptation to zero-shot and few-shot learning. In this work, we address this problem by transfer clustering that aims to learn a discriminative latent space of the unlabelled target data in a novel domain by knowledge transfer from labelled related domains. Specifically, we want to leverage relative (pairwise) imagery information, which is freely available and intrinsic to a target domain, to model the target domain image distribution characteristics as well as the prior-knowledge learned from related labelled domains to enable more discriminative clustering of unlabelled target data. Our method mitigates nontransferrable prior-knowledge by self-supervision, benefiting from both transfer and self-supervised learning. Extensive experiments on four datasets for image clustering tasks reveal the superiority of our model over the state-of-the-art transfer clustering techniques. We further demonstrate its competitive transferability on four zero-shot learning benchmarks.

Motivation & Objective

• Address the challenge of transferring knowledge from labeled source domains to unlabelled target domains with unknown and shifted data distributions.
• Overcome limitations of existing unsupervised transfer learning methods that rely on strong assumptions, such as shared label spaces or semantic prototypes.
• Develop a scalable, generic solution for transfer clustering that does not require word vectors or attribute annotations.
• Improve discriminative clustering of unlabelled target data by jointly leveraging transferred supervision and self-supervised constraints from intrinsic image relationships.

Proposed method

• Introduce a self-supervised remedy (SUPREME) that combines transferred supervision from a pre-trained source model with self-supervised learning based on pairwise image similarities in the target domain.
• Use a contrastive learning objective to generate self-supervision by comparing image pairs in the target domain, modeling intrinsic distribution characteristics.
• Apply a temperature-controlled confidence distribution to balance the reliability of transferred prior-knowledge and self-supervised signals.
• Incorporate multiple regularizations: balance regularization to stabilize cluster size distribution, binary regularization to preserve shared features, and multi-task regularization to prevent trivial representations.
• Train the model end-to-end using a joint objective that optimizes both transferred and self-supervised signals in a unified latent space.
• Leverage the auxiliary learning principle to allow self-supervision to correct for nontransferrable or misleading prior-knowledge in the presence of domain shift.

Experimental results

Research questions

• RQ1 Can self-supervised learning effectively complement transferred supervision to improve clustering in unlabelled target domains with distribution shift?
• RQ2 How does the proposed SUPREME method perform compared to state-of-the-art transfer clustering and zero-shot learning models without using semantic or attribute-based supervision?
• RQ3 What is the relative contribution of transferred supervision versus self-supervision in the model’s performance, and how do they interact?
• RQ4 How robust is the SUPREME model to hyperparameter choices such as the temperature parameter in the confidence distribution?
• RQ5 To what extent can SUPREME generalize to zero-shot learning benchmarks without relying on word vector spaces or attribute learning?

Key findings

• SUPREME achieves state-of-the-art performance on four transfer clustering benchmarks by effectively combining transferred and self-supervised signals.
• The model demonstrates competitive performance on four zero-shot learning benchmarks without using any textual or attribute-based semantic representations.
• Ablation studies show that both transferred supervision and self-supervision independently contribute to performance, with their combination yielding the best results.
• The temperature hyperparameter has a significant but dataset-dependent effect on performance, indicating sensitivity to domain-specific distribution shifts.
• Removal of any of the three regularizations (balance, binary, multi-task) leads to a significant performance drop, confirming their necessity and non-redundancy.
• On CIFAR10, the self-supervised model alone marginally outperforms SUPREME, suggesting that prior-knowledge can sometimes be misleading or nontransferrable in certain domains.