[Paper Review] A critical analysis of self-supervision, or what we can learn from a single image

Yuki M. Asano, Christian Rupprecht+1 more|arXiv (Cornell University)|Apr 30, 2019

Generative Adversarial Networks and Image Synthesis67 references65 citations

TL;DR

The paper analyzes self-supervised representations from different depths on ImageNet and shows how linear probes behave for varying patch-based augmentations, highlighting that self-supervised methods lag supervised performance at deeper layers but can match or exceed at early layers depending on training schemes and patch sizes.

ABSTRACT

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Motivation & Objective

• Evaluate how self-supervised representations perform across network depths using linear probes on ImageNet.
• Investigate the effect of patch-based augmentation strategies on learning low-level versus high-level features.
• Compare self-supervised methods to supervised baselines to identify depth-dependent gaps in performance.

Proposed method

• Train self-supervised models with various augmentation strategies including small patches.
• Evaluate representations with linear classifiers attached to different network layers.
• Plot absolute and relative (to supervised) accuracies across layers to assess depth-dependent trends.
• Analyze how training with small patches emphasizes learning low-level filters (conv1).
• Compare performance degradation of unsupervised models versus supervised baselines with depth.

Experimental results

Research questions

• RQ1 How do self-supervised representations perform at different network depths on ImageNet when evaluated with linear probes?
• RQ2 Does patch size in augmentation influence the learned representations, particularly low-level filters?
• RQ3 What is the depth-dependent gap between self-supervised methods and supervised baselines on ImageNet?
• RQ4 Can self-supervised methods match or exceed supervised performance at early layers under certain training schemes?
• RQ5 How do different augmentation strategies affect transferability of features across layers?

Key findings

• Self-supervised models can outperform supervised models at the very first conv layer under certain patch-based training schemes.
• Performance of self-supervised methods generally degrades with depth beyond conv3, while supervised models improve with depth.
• Relative gains of self-supervised methods over supervision vary by layer, with a widening gap at deeper layers due to task-specific supervision.
• Patch-based augmentation with very small patches emphasizes learning useful low-level filters.
• Linear probes on representations from self-supervised models show comparable performance to supervised baselines on early layers under specific training regimes.