[Paper Review] The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models

Tianlong Chen, Jonathan Frankle+5 more|arXiv (Cornell University)|Dec 12, 2020

Advanced Neural Network ApplicationsComputer Science87 references31 citations

TL;DR

This paper investigates whether matching subnetworks exist within pre-trained computer vision models (supervised and self-supervised) that can transfer to diverse downstream tasks without performance loss. It shows universal transferable tickets at substantial sparsities across classification, detection, and segmentation.

ABSTRACT

The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervised pre-training such as simCLR and MoCo. Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Latest studies suggest that pre-training benefits from gigantic model capacity. We are hereby curious and ask: after pre-training, does a pre-trained model indeed have to stay large for its downstream transferability? In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH). LTH identifies highly sparse matching subnetworks that can be trained in isolation from (nearly) scratch yet still reach the full models' performance. We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance. Our extensive experiments convey an overall positive message: from all pre-trained weights obtained by ImageNet classification, simCLR, and MoCo, we are consistently able to locate such matching subnetworks at 59.04% to 96.48% sparsity that transfer universally to multiple downstream tasks, whose performance see no degradation compared to using full pre-trained weights. Further analyses reveal that subnetworks found from different pre-training tend to yield diverse mask structures and perturbation sensitivities. We conclude that the core LTH observations remain generally relevant in the pre-training paradigm of computer vision, but more delicate discussions are needed in some cases. Codes and pre-trained models will be made available at: https://github.com/VITA-Group/CV_LTH_Pre-training.

Motivation & Objective

• Assess whether matching subnetworks exist in pre-trained CV models that preserve downstream transfer performance.
• Determine if there are universal subnetworks transferable across diverse downstream tasks (classification, detection, segmentation).
• Compare subnetworks derived from supervised versus self-supervised pre-training in terms of transferability and structure sensitivity.

Proposed method

• Treat pre-trained weights as initialization for subnetworks.
• Apply iterative magnitude pruning (IMP) to identify matching subnetworks.
• Define matching subnetworks as those whose transfer performance is at least as good as the full pre-trained model under the same training regime.
• Evaluate transferability of subnetworks across multiple downstream tasks and datasets (classification, detection, segmentation).
• Analyze mask diversity and perturbation sensitivities across pre-training types (ImageNet, simCLR, MoCo).
• Explore effects of larger pre-trained models and temperature settings on transferability.

Experimental results

Research questions

• RQ1 Do winning tickets found on pre-training tasks also serve as winning tickets for downstream tasks?
• RQ2 Are there universal, transferable subnetworks across diverse downstream tasks when initialized from different pre-training schemes?
• RQ3 How do subnetworks from supervised versus self-supervised pre-training compare in transferability and mask structure?

Key findings

• Winning tickets exist with 67.23%, 59.04%, and 95.60% sparsity for supervised ImageNet, simCLR, and MoCo pre-training, respectively.
• Subnetworks from pre-training transfer universally to diverse downstream classification tasks at sparsities around 86.58%–91.41% for CIFAR-10, CIFAR-100, SVHN, and Fashion-MNIST, with VisDA2017 requiring higher capacity (around 67.23%–59.04%).
• Subnetworks transferred from pre-training can outperform subnetworks found directly on downstream tasks (e.g., 95.60%/93.13%/97.75% sparsity for detection and segmentation).
• Subnetworks from MoCo pre-training provide best transfer to detection/segmentation among pre-training types, while ImageNet and simCLR show different strengths depending on downstream task and sparsity.
• Subnetworks identified from pre-training exhibit diverse mask structures and perturbation sensitivities, with less than 6.55% mask overlap after five IMP rounds across pre-training types.
• Pruning larger pre-trained models yields better transferable subnetworks for self-supervised pre-training (simCLR) as shown by CIFAR-100 results when comparing ResNet-50 vs ResNet-152.