[Paper Review] Simple Distillation Baselines for Improving Small Self-supervised Models

Jindong Gu, Wei Liu+1 more|arXiv (Cornell University)|Jun 21, 2021

Domain Adaptation and Few-Shot Learning14 references4 citations

TL;DR

This paper proposes SimDis, a simple knowledge distillation framework to improve small self-supervised models using either offline or online distillation. By distilling representations from a large pre-trained teacher model into a small student model, SimDis achieves state-of-the-art linear evaluation accuracy on ResNet-18, outperforming prior methods like SEED without requiring memory banks or complex components.

ABSTRACT

While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation baseline, which establishes a new state-of-the-art, and an online-distillation baseline, which achieves similar performance with minimal computational overhead. We hope these baselines will provide useful experience for relevant future research. Code is available at: https://github.com/JindongGu/SimDis/

Motivation & Objective

• Address the performance gap between small self-supervised models and their supervised counterparts, which widens significantly as model size decreases.
• Develop practical and efficient distillation baselines that improve small self-supervised models without relying on complex components like memory banks or contrastive negatives.
• Demonstrate that distillation from a large teacher model can significantly boost the performance of small student models in self-supervised learning.
• Explore the effectiveness of distillation under minimal training settings, such as without LARS optimizer or SyncBN, to improve practicality and reduce computational cost.

Proposed method

• Proposes an offline distillation baseline (SimDis-Off) where a large teacher model is first pre-trained using BYOL, then used to supervise a small student model.
• Introduces an online distillation baseline (SimDis-On) where the student and teacher models are trained simultaneously, with the student learning from multiple views of the teacher's representations.
• Uses a siamese network architecture for both teacher and student, with the student trained to predict normalized projections from both its own and the teacher's networks.
• Employs mean squared error loss between normalized predictions from the student and normalized projections from the teacher, following the BYOL protocol.
• Enables multi-view distillation by combining multiple target views from the teacher and student networks to improve optimization stability and performance.
• Removes dependency on LARS optimizer and SyncBN by training on a single machine with 8 GPUs, demonstrating robustness under simpler, more practical settings.

Experimental results

Research questions

• RQ1 Can simple knowledge distillation from a large pre-trained teacher model significantly improve the performance of small self-supervised student models?
• RQ2 How does online distillation compare to offline distillation in terms of performance and computational cost when training small models?
• RQ3 To what extent can distillation improve small self-supervised models when critical components like LARS and SyncBN are removed?
• RQ4 Does multi-view distillation from both student and teacher networks enhance learning efficiency and final accuracy?
• RQ5 Can distillation baselines achieve state-of-the-art performance without using memory banks or negative samples, as in prior methods like SEED?

Key findings

• SimDis-Off achieves 67.18% top-1 accuracy on ImageNet linear evaluation with ResNet-18 after 1000 epochs, outperforming previous SOTA methods like SEED.
• SimDis-On achieves 66.78% top-1 accuracy under the same setting, demonstrating that online distillation can match offline performance with minimal computational overhead.
• The performance gap between offline and online distillation narrows significantly with longer training: from 3.65% at 100 epochs to less than 1% at 1000 epochs.
• Multi-view distillation (SimDis-On-7v) improves online distillation by 2.57% over single-view, narrowing the gap with offline distillation while adding negligible FLOPs.
• In settings without LARS or SyncBN, SimDis-Off achieves 62.66% top-1 accuracy and SimDis-On achieves 63.52% with all 7 views, showing strong performance under minimal, practical training conditions.
• The student model achieves 55.96% top-1 accuracy without distillation, but improves to 63.52% with full multi-view distillation, demonstrating the effectiveness of the proposed distillation strategy.