[Paper Review] Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey

Longlong Jing, Yingli Tian|arXiv (Cornell University)|Feb 16, 2019

Advanced Image and Video Retrieval Techniques164 references177 citations

TL;DR

This paper provides a comprehensive survey of deep ConvNet-based self-supervised visual feature learning, detailing architectures, pretext tasks, datasets, evaluation, and future directions.

ABSTRACT

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the main components and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used image and video datasets and the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.

Motivation & Objective

• Motivate the use of self-supervised learning to learn visual features from large-scale unlabeled data.
• Review network architectures and common pretext tasks used in self-supervised visual feature learning.
• Summarize datasets, evaluation protocols, and downstream tasks used to assess learned features.
• Provide quantitative performance comparisons and discuss promising future directions.

Proposed method

• Describe the general self-supervised learning pipeline where a ConvNet is trained on a pretext task using automatically generated pseudo labels and then transferred to downstream tasks.
• Classify learning schemes (supervised, semi-supervised, weakly supervised, unsupervised, with emphasis on self-supervised) and formalize their loss objectives.
• Categorize pretext tasks into generation-based, context-based, free semantic label-based, and cross-modal based on data attributes used for supervision.
• Outline common image and video architectures (AlexNet, VGG, GoogLeNet, ResNet, DenseNet; 2D/3D ConvNets; LSTM-based models) and their roles in feature learning.
• Explain evaluation via downstream tasks such as image classification, semantic segmentation, object detection, and human action recognition, plus qualitative visualization.
• Summarize commonly used image/video datasets and discuss how pretext tasks drive learned feature quality.

Experimental results

Research questions

• RQ1 What pretext tasks and architectural choices yield transferable, high-quality visual features in self-supervised learning?
• RQ2 How do self-supervised features compare across downstream tasks like image classification, segmentation, detection, and action recognition?
• RQ3 What are effective strategies for evaluating and benchmarking self-supervised visual feature learning methods?
• RQ4 What future directions can bridge the gap between self-supervised and supervised performance on visual tasks?

Key findings

• Self-supervised methods can learn transferable visual features from large unlabeled datasets without human annotations.
• Pretext tasks are categorized into generation-based, context-based, free semantic label-based, and cross-modal-based, each guiding feature learning.
• Common downstream evaluations include image classification, semantic segmentation, object detection, and action recognition to assess feature generalization.
• Pre-trained self-supervised models on large-scale data can speed up training and improve downstream performance, narrowing the gap with supervised methods.
• The paper provides a quantitative performance comparison across methods and datasets, highlighting trends and areas for improvement.