[Paper Review] Pruning Convolutional Neural Networks with Self-Supervision

Mathilde Caron, Ari S. Morcos+3 more|arXiv (Cornell University)|Jan 10, 2020

Domain Adaptation and Few-Shot LearningComputer Science43 references31 citations

TL;DR

The paper studies using standard magnitude-based unstructured pruning, typically used for supervised learning, on networks trained with self-supervised tasks, showing pruned self-supervised subnetworks preserve transfer performance and can be re-trained on labels.

ABSTRACT

Convolutional neural networks trained without supervision come close to matching performance with supervised pre-training, but sometimes at the cost of an even higher number of parameters. Extracting subnetworks from these large unsupervised convnets with preserved performance is of particular interest to make them less computationally intensive. Typical pruning methods operate during training on a task while trying to maintain the performance of the pruned network on the same task. However, in self-supervised feature learning, the training objective is agnostic on the representation transferability to downstream tasks. Thus, preserving performance for this objective does not ensure that the pruned subnetwork remains effective for solving downstream tasks. In this work, we investigate the use of standard pruning methods, developed primarily for supervised learning, for networks trained without labels (i.e. on self-supervised tasks). We show that pruned masks obtained with or without labels reach comparable performance when re-trained on labels, suggesting that pruning operates similarly for self-supervised and supervised learning. Interestingly, we also find that pruning preserves the transfer performance of self-supervised subnetwork representations.

Motivation & Objective

• Assess whether standard pruning methods developed for supervised learning work on self-supervised networks.
• Evaluate how pruning affects transferability of self-supervised features to downstream tasks.
• Investigate whether pruned self-supervised subnetworks can be re-trained on supervised tasks.
• Examine the impact of using some supervision during pruning (semi-supervised pruning) on pruning outcomes.

Proposed method

• Apply magnitude-based unstructured iterative pruning to over-parameterized networks trained with self-supervised tasks.
• Use weight resetting strategies (winning tickets) or random re-initialization when retraining pruned subnetworks.
• Evaluate two self-supervised tasks: RotNet and Exemplar (plus NPDI) as pretext objectives for pruning.
• Transfer pruned representations to downstream tasks (VOC07, Places205, ImageNet) using linear/finetuned classifiers.
• Compare pruning with supervision, without supervision, and semi-supervised pruning (partial labels) during pruning.
• Report results across pruning rates from 20% to 99.9% remaining weights.

Experimental results

Research questions

• RQ1 Can standard supervised pruning methods produce effective subnetworks when the base network is trained with self-supervised objectives?
• RQ2 Do pruned self-supervised subnetworks retain transfer performance on downstream tasks comparable to pruning with labels?
• RQ3 Are pruned subnetworks re-trainable on supervised tasks from their initialization, and how do winning tickets compare to random initializations?
• RQ4 Does incorporating some label supervision during pruning (semi-supervised pruning) improve the quality of the resulting subnetworks?

Key findings

• Pruning up to 90% of weights (less than 0.1% remaining) does not deteriorate transfer performance when evaluated with linear classifiers on VOC07, Places205, and ImageNet.
• Pruned self-supervised subnetworks can be re-trained to match or exceed supervised pruning performance when randomly re-initialized, while winning tickets are less advantageous for self-supervised pruning.
• Transferring pruned networks tends to yield better results when pruning is done before transfer rather than during transfer; pruned pre-trained models can be released for users with limited compute.
• Self-supervised pruning yields masks whose quality is similar to supervised pruning, but the associated winning-ticket initializations are not as strong as those inherited from label classification.
• Semi-supervised pruning (using ~10% labeled data) improves winning-ticket initializations and downstream accuracy over unsupervised pruning, indicating some label-dependence of the winning tickets.