[Paper Review] TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

Kan Wu, Houwen Peng+11 more|arXiv (Cornell University)|Sep 21, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

TinyCLIP proposes a novel cross-modal distillation method that leverages affinity mimicking and weight inheritance to compress large-scale vision-language models like CLIP. By mimicking cross-modal feature alignment in an affinity space and inheriting pre-trained weights from teacher to student models, TinyCLIP achieves state-of-the-art zero-shot performance—surpassing the original CLIP ViT-B/16 with only 8.9% of its parameters (8.9M parameters) while maintaining 41.1% top-1 accuracy on ImageNet.

ABSTRACT

In this paper, we propose a novel cross-modal distillation method, called TinyCLIP, for large-scale language-image pre-trained models. The method introduces two core techniques: affinity mimicking and weight inheritance. Affinity mimicking explores the interaction between modalities during distillation, enabling student models to mimic teachers' behavior of learning cross-modal feature alignment in a visual-linguistic affinity space. Weight inheritance transmits the pre-trained weights from the teacher models to their student counterparts to improve distillation efficiency. Moreover, we extend the method into a multi-stage progressive distillation to mitigate the loss of informative weights during extreme compression. Comprehensive experiments demonstrate the efficacy of TinyCLIP, showing that it can reduce the size of the pre-trained CLIP ViT-B/32 by 50%, while maintaining comparable zero-shot performance. While aiming for comparable performance, distillation with weight inheritance can speed up the training by 1.4 - 7.8 $ imes$ compared to training from scratch. Moreover, our TinyCLIP ViT-8M/16, trained on YFCC-15M, achieves an impressive zero-shot top-1 accuracy of 41.1% on ImageNet, surpassing the original CLIP ViT-B/16 by 3.5% while utilizing only 8.9% parameters. Finally, we demonstrate the good transferability of TinyCLIP in various downstream tasks. Code and models will be open-sourced at https://aka.ms/tinyclip.

Motivation & Objective

• Address the high computational and storage costs of large-scale vision-language models like CLIP by enabling efficient model distillation.
• Overcome the limitations of single-modal distillation in multi-modal settings by preserving cross-modal interactions during knowledge transfer.
• Improve distillation efficiency by leveraging pre-trained weights from teacher models through a novel weight inheritance mechanism.
• Enable extreme model compression without significant performance drop by introducing multi-stage progressive distillation.
• Enhance transferability of distilled models across zero-shot and downstream vision-language tasks.

Proposed method

• Affinity mimicking trains the student model to replicate the visual-linguistic feature alignment behavior of the teacher by minimizing the cosine similarity gap between image and text embeddings in the affinity space.
• Weight inheritance transfers pre-trained weights from the teacher model to the student, using either manual selection of k-dimension or k-layer weights or learnable masks to identify important parameters.
• Learnable masks are applied independently to vision and text branches to automatically identify and inherit the most informative weights, adapting to modality-specific importance.
• Multi-stage progressive distillation breaks down extreme compression into sequential stages, where each student inherits weights from a similarly structured teacher, reducing architectural mismatch and improving performance.
• The method uses a two-branch architecture with shared training objectives: contrastive loss on image-text pairs and knowledge distillation via affinity mimicking.
• Distillation is optimized using a combination of cross-entropy loss and alignment loss, with the teacher’s output affinity matrix serving as a supervisory signal for the student.

Experimental results

Research questions

• RQ1 How can cross-modal distillation be improved by explicitly modeling the interaction between vision and language modalities during knowledge transfer?
• RQ2 What impact does weight inheritance from a pre-trained teacher model have on distillation efficiency and final performance in vision-language models?
• RQ3 Can multi-stage progressive distillation mitigate performance degradation during extreme model compression in vision-language models?
• RQ4 How does affinity mimicking compare to conventional distillation methods that rely only on image or text features?
• RQ5 Does the distilled TinyCLIP model maintain strong zero-shot and fine-tuning performance across diverse downstream vision-language benchmarks?

Key findings

• TinyCLIP reduces the size of CLIP ViT-B/32 by 50% while maintaining comparable zero-shot performance on ImageNet.
• Training with weight inheritance accelerates distillation by 1.4× to 7.8× compared to training from scratch, significantly improving training efficiency.
• TinyCLIP ViT-8M/16 achieves 41.1% zero-shot top-1 accuracy on ImageNet—surpassing the original CLIP ViT-B/16 (37.6%)—while using only 8.9% of its parameters.
• TinyCLIP ViT-63M/32 achieves 55.7% top-1 accuracy on ImageNet-V2, outperforming OpenCLIP ViT-B/32 (55.1%) and CLIP ViT-B/32 (56.0%) on robustness benchmarks.
• The multi-stage progressive distillation strategy improves zero-shot accuracy by 2.0% on ImageNet compared to single-stage distillation, demonstrating its effectiveness in preserving knowledge.
• TinyCLIP models show strong transferability, achieving top-1 accuracy within 2% of OpenCLIP ViT-B/32 on 23 downstream datasets in both zero-shot and linear-probe evaluations.