[Paper Review] VeCLIP: Improving CLIP Training via Visual-enriched Captions

Zhengfeng Lai, Haotian Zhang+10 more|arXiv (Cornell University)|Oct 11, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

VeCLIP improves CLIP pre-training by generating visual-enriched captions (VeC) using LLaVA and a second LLM to enhance image-text alignment, while employing a mixed training strategy alternating between original AltTexts and VeC. This approach achieves over 20% improvement on COCO and Flickr30k retrieval under 12M data, and over 3% accuracy gain using only 14% of vanilla CLIP's data.

ABSTRACT

Large-scale web-crawled datasets are fundamental for the success of pre-training vision-language models, such as CLIP. However, the inherent noise and potential irrelevance of web-crawled AltTexts pose challenges in achieving precise image-text alignment. Existing methods utilizing large language models (LLMs) for caption rewriting have shown promise on small, curated datasets like CC3M and CC12M. This study introduces a scalable pipeline for noisy caption rewriting. Unlike recent LLM rewriting techniques, we emphasize the incorporation of visual concepts into captions, termed as Visual-enriched Captions (VeCap). To ensure data diversity, we propose a novel mixed training scheme that optimizes the utilization of AltTexts alongside newly generated VeCap. We showcase the adaptation of this method for training CLIP on large-scale web-crawled datasets, termed VeCLIP. Employing this cost-effective pipeline, we effortlessly scale our dataset up to 300 million samples named VeCap dataset. Our results show significant advantages in image-text alignment and overall model performance. For example, VeCLIP achieves up to +25.2% gain in COCO and Flickr30k retrieval tasks under the 12M setting. For data efficiency, VeCLIP achieves +3% gain while only using 14% of the data employed in the vanilla CLIP and 11% in ALIGN. We also note the VeCap data is complementary with other well curated datasets good for zero-shot classification tasks. When combining VeCap and DFN, our model can achieve strong performance on both of image-text retrieval and zero-shot classification tasks, e.g. 83.1% accuracy@1 on ImageNet zero-shot for a H/14 model. We release the pre-trained models at https://github.com/apple/ml-veclip.

Motivation & Objective

• To address the scarcity of high-quality, visually grounded captions in large-scale web-crawled datasets used for CLIP pre-training.
• To improve image-text alignment by injecting visual concepts into captions without relying on pre-existing high-quality annotations.
• To increase data diversity during pre-training by alternating between original AltTexts and LLM-generated visual-enriched captions (VeC).
• To enable efficient, scalable pre-training of vision-language models on massive, noisy web-crawled data without filtering out low-quality pairs.
• To demonstrate that VeCLIP achieves superior performance with significantly less data compared to vanilla CLIP and ALIGN.

Proposed method

• Use LLaVA to detect visual entities in images and generate structured visual descriptions.
• Feed the detected visual entities and the original AltText into a second LLM to generate enriched captions (VeC) that integrate visual concepts.
• Construct a mixed training scheme that alternates between original AltTexts and LLM-generated VeC during pre-training to increase data diversity.
• Retain all image-text pairs, including those with noisy AltTexts, to preserve data efficiency and avoid data loss from filtering.
• Adapt the CLIP contrastive loss objective to train the vision and text encoders using both original and VeC-modified captions.
• Use a two-stage captioning pipeline: first, visual grounding via LLaVA; second, natural language refinement via a large language model to produce fluent, concept-rich captions.

Experimental results

Research questions

• RQ1 Can injecting visual concepts into captions via LLMs significantly improve image-text alignment in CLIP pre-training?
• RQ2 Does using a mixed training strategy that alternates between original AltTexts and visual-enriched captions (VeC) enhance model generalization and reduce overfitting?
• RQ3 To what extent can VeCLIP reduce data requirements while maintaining or improving performance compared to vanilla CLIP and ALIGN?
• RQ4 How does the quality of captions affect downstream zero-shot image classification and retrieval performance?
• RQ5 Can the VeCLIP framework be generalized to well-curated datasets like CC3M and CC12M, and does it yield consistent improvements?

Key findings

• VeCLIP achieves a 20.1% improvement in COCO image-to-text retrieval and a 21.3% improvement in Flickr30k retrieval under the 12M data setting.
• On ImageNet zero-shot classification, VeCLIP improves accuracy by over 3% when using only 14% of the data required by vanilla CLIP.
• Using 11% of the data used in ALIGN, VeCLIP achieves a 3.1% accuracy gain on ImageNet zero-shot classification.
• The mixed training strategy that alternates between AltTexts and VeC improves zero-shot ImageNet performance by 1.8% compared to using only VeC.
• On the CC3M dataset, LLM-VeC improves COCO image-to-text retrieval by 18.16% and ImageNet zero-shot accuracy by 6.14%.
• VeCLIP consistently outperforms vanilla CLIP across all backbone sizes, including ViT-L/14, demonstrating scalability and architecture independence.