[Paper Review] Revisiting Self-Supervised Visual Representation Learning

Alexander Kolesnikov, Xiaohua Zhai+1 more|arXiv (Cornell University)|Jan 25, 2019

Domain Adaptation and Few-Shot Learning49 references68 citations

TL;DR

This paper conducts a large-scale study of self-supervised visual representation learning, showing that CNN architecture choice and width profoundly affect learned representations, and achieving new state-of-the-art results by optimizing architectures and pretext tasks together.

ABSTRACT

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in selfsupervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.

Motivation & Objective

• Assess how CNN architecture choices influence quality of self-supervised visual representations.
• Determine whether standard supervised-design practices transfer to self-supervised settings.
• Identify how network width and invertibility affect representation quality.
• Evaluate the adequacy of linear evaluation for assessing representations learned via self-supervision.
• Provide guidance for selecting architectures and tasks to boost unsupervised learning performance.

Proposed method

• Evaluate six CNN architectures (ResNet variants, RevNet, VGG) with different width factors (k) on self-supervised tasks.
• Revisit four self-supervised techniques (Rotation, Exemplar, Relative Patch Location, Jigsaw) across architectures.
• Use pre-logits representations to train a linear logistic regression classifier for downstream ImageNet/Places205 tasks.
• Compare linear vs. non-linear (MLP) evaluation to assess adequacy of linear probes.
• Analyze the effect of network width and representation size independently.
• Examine SGD training dynamics for linear evaluation to understand convergence requirements.

Experimental results

Research questions

• RQ1 How does CNN architecture influence the quality of representations learned through self-supervised tasks?
• RQ2 Do standard supervised-design CNN choices translate to the self-supervised setting?
• RQ3 What is the impact of increasing network width and representation size on self-supervised performance?
• RQ4 Is linear evaluation sufficient to gauge representation quality across architectures and tasks?
• RQ5 How do skip-connections and invertibility affect the preservation of useful representations in deep networks?

Key findings

• Architecture choice markedly affects self-supervised performance and rankings vary across tasks.
• Skip-connections help prevent degradation of representation quality toward deeper layers in self-supervised learning.
• Increasing the number of filters (width) and representation size consistently improves performance.
• Linear evaluation is largely adequate; non-linear evaluation yields only marginal gains in this context.
• Context prediction originally sparked self-supervised learning and, with proper architecture, can achieve leading results.
• Wider models provide benefits across datasets (ImageNet and Places205) and across low-data regimes.