[Paper Review] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Wenliang Dai, Lu Hou+4 more|arXiv (Cornell University)|Mar 12, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Vision-Language Knowledge Distillation (VLKD), a data- and computation-efficient method to enable CLIP for multimodal generation by distilling knowledge from a pre-trained BART language model into CLIP's vision-text space. The approach achieves state-of-the-art zero-shot performance—44.5% accuracy on VQAv2 and 84.6 CIDEr on COCO captioning—while preserving BART’s original language understanding and generation capabilities.

ABSTRACT

The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous state-of-the-art zero-shot model with $7 imes$ fewer parameters. Furthermore, the original textual language understanding and generation ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.

Motivation & Objective

• To address the limitation of dual-stream VLP models like CLIP, which lack generative capabilities due to a weak text encoder.
• To enable CLIP to perform zero-shot multimodal generation tasks such as visual question answering and image captioning without fine-tuning.
• To maintain the original language understanding and generation performance of the pre-trained language model (BART) after distillation.
• To develop a method that is both data- and computation-efficient compared to pre-training from scratch.
• To demonstrate that knowledge distillation can effectively transfer multimodal understanding from CLIP to a strong PLM while preserving unimodal capabilities.

Proposed method

• Freeze CLIP’s vision and text encoders to preserve their pre-trained multimodal representations during distillation.
• Align the BART encoder with CLIP’s joint multimodal embedding space via contrastive learning to enable multimodal understanding.
• Apply an image-conditioned language modeling loss to couple BART’s encoder and decoder for generative tasks.
• Use a masked language modeling objective with variable-length masks to train the model to generate coherent captions from visual inputs.
• Perform knowledge distillation using a small subset of image-text pairs (e.g., 1M or 3M) to reduce computational cost.
• Discard CLIP’s original text encoder at inference, replacing it with the distilled BART for both multimodal and unimodal tasks.

Experimental results

Research questions

• RQ1 Can knowledge distillation effectively transfer multimodal generation capabilities to CLIP without fine-tuning?
• RQ2 How does the performance of the distilled model compare to state-of-the-art zero-shot models on multimodal generation benchmarks?
• RQ3 To what extent is the original language understanding and generation ability of BART preserved after distillation?
• RQ4 What is the impact of different distillation objectives (e.g., TTDM, ITCL) on model performance?
• RQ5 How sensitive is the method to the size of the distillation dataset and the number of masked tokens?

Key findings

• The VLKD model achieves 44.5% zero-shot accuracy on the VQAv2 dataset, surpassing the previous SOTA zero-shot model by a significant margin and using 7× fewer parameters.
• On the COCO image captioning benchmark, the model achieves 84.6 CIDEr in zero-shot evaluation, demonstrating strong generative capability without fine-tuning.
• The model maintains near-identical performance to the original BART on the GLUE benchmark, indicating minimal degradation of NLU abilities.
• Abllation studies show that both TTDM and ITCL distillation objectives are critical, with removing both causing a 14.5-point drop in VQAv2 accuracy and a 13.5-point drop in CIDEr.
• Performance degrades sharply when the distillation dataset is reduced to 100K samples, indicating that sufficient data is essential for effective knowledge transfer.
• Unfreezing CLIP’s weights during distillation leads to a significant performance drop (31.7% on VQAv2), confirming that freezing preserves the pre-trained multimodal space.