[Paper Review] Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms

Xingyi Yang, Xuehai He+4 more|arXiv (Cornell University)|Jun 19, 2020

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper conducts a comprehensive empirical comparison of self-supervised learning (SSL) and transfer learning (TL) across multiple data and task properties. It identifies that SSL outperforms TL when domain shift is large or pretraining data is limited, while TL excels with large, balanced source datasets and small domain gaps. The study provides actionable guidelines for selecting pretraining methods based on application-specific characteristics.

ABSTRACT

Pretraining has become a standard technique in computer vision and natural language processing, which usually helps to improve performance substantially. Previously, the most dominant pretraining method is transfer learning (TL), which uses labeled data to learn a good representation network. Recently, a new pretraining approach -- self-supervised learning (SSL) -- has demonstrated promising results on a wide range of applications. SSL does not require annotated labels. It is purely conducted on input data by solving auxiliary tasks defined on the input data examples. The current reported results show that in certain applications, SSL outperforms TL and the other way around in other applications. There has not been a clear understanding on what properties of data and tasks render one approach outperforms the other. Without an informed guideline, ML researchers have to try both methods to find out which one is better empirically. It is usually time-consuming to do so. In this work, we aim to address this problem. We perform a comprehensive comparative study between SSL and TL regarding which one works better under different properties of data and tasks, including domain difference between source and target tasks, the amount of pretraining data, class imbalance in source data, and usage of target data for additional pretraining, etc. The insights distilled from our comparative studies can help ML researchers decide which method to use based on the properties of their applications.

Motivation & Objective

• To address the lack of clear guidelines for choosing between self-supervised learning (SSL) and transfer learning (TL) in practice.
• To investigate how data and task properties—such as domain difference, pretraining data size, class imbalance, and target data usage—affect the relative performance of SSL and TL.
• To provide ML researchers with data-driven, actionable recommendations for selecting the optimal pretraining paradigm based on application-specific conditions.
• To reduce the need for empirical trial-and-error by identifying systematic factors that favor one method over the other.

Proposed method

• Conducted controlled experiments on 5 source tasks and 4 target tasks spanning diverse domains, including natural images, medical imaging, and scene recognition.
• Systematically varied four key factors: domain difference between source and target tasks, amount of pretraining data, class imbalance in source data, and use of target data for additional pretraining.
• Used standard evaluation protocols: pretrain on source data, then fine-tune on target data, measuring accuracy across all combinations.
• Compared SSL (e.g., MoCo-style contrastive learning) and TL (supervised pretraining) under identical network architectures and training procedures.
• Evaluated performance stability across varying data conditions, including extreme class imbalance and domain shift.
• Analyzed the impact of pretraining on combined source + target data, comparing it to source-only and target-only pretraining.

Experimental results

Research questions

• RQ1 Under what conditions does self-supervised learning (SSL) outperform transfer learning (TL), and vice versa?
• RQ2 How does domain difference between source and target tasks affect the relative performance of SSL and TL?
• RQ3 How does the amount of pretraining data influence the comparative effectiveness of SSL versus TL?
• RQ4 How sensitive are SSL and TL to class imbalance in the source dataset?
• RQ5 Does using target task data for additional pretraining improve performance, and does this effect differ between SSL and TL?

Key findings

• When domain difference between source and target tasks is large, SSL outperforms TL; when domain difference is small, TL performs better.
• With small amounts of pretraining data, SSL outperforms TL; with large amounts, TL surpasses SSL.
• SSL performance is more stable across varying domain differences compared to TL, which shows significant performance drops under high domain shift.
• For small domain differences, SSL is less sensitive to pretraining data size than TL, which shows large performance fluctuations with data quantity changes.
• SSL is more robust to class imbalance: the performance gap across imbalance ratios is 4.3% (absolute) for SSL versus 6.5% for TL on the COVID-CT task.
• For SSL, pretraining on combined source and target data improves performance over source-only or target-only; for TL, combined pretraining underperforms source-only due to reduced focus on the target task.