[Paper Review] Retrieval-Enhanced Contrastive Vision-Text Models

Ahmet İşcen, Mathilde Caron+2 more|arXiv (Cornell University)|Jun 12, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Retrieval-Enhanced Contrastive (RECO) models that improve zero-shot fine-grained vision-language understanding by fusing CLIP embeddings with cross-modal knowledge retrieved from an external memory. Using uni-modal retrieval (image-to-image or text-to-text) followed by cross-modal fusion, RECO boosts performance on challenging benchmarks like Stanford Cars (+10.9), CUB-2011 (+10.2), and OVEN (+7.3) without fine-tuning.

ABSTRACT

Contrastive image-text models such as CLIP form the building blocks of many state-of-the-art systems. While they excel at recognizing common generic concepts, they still struggle on fine-grained entities which are rare, or even absent from the pre-training dataset. Hence, a key ingredient to their success has been the use of large-scale curated pre-training data aiming at expanding the set of concepts that they can memorize during the pre-training stage. In this work, we explore an alternative to encoding fine-grained knowledge directly into the model's parameters: we instead train the model to retrieve this knowledge from an external memory. Specifically, we propose to equip existing vision-text models with the ability to refine their embedding with cross-modal retrieved information from a memory at inference time, which greatly improves their zero-shot predictions. Remarkably, we show that this can be done with a light-weight, single-layer, fusion transformer on top of a frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive (RECO) training improves CLIP performance substantially on several challenging fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and +7.3 on the recent OVEN benchmark, where we even outperform the fine-tuned models on unseen classes.

Motivation & Objective

• To improve zero-shot fine-grained image and text classification beyond what is achievable with standard contrastive vision-language models like CLIP.
• To explore whether external knowledge retrieval can enhance model generalization without retraining or fine-tuning.
• To address the limitation of CLIP in recognizing rare or fine-grained entities due to insufficient coverage in pre-training data.
• To design a lightweight, generic, and deployable method that enhances model representations at inference time using a large-scale external memory.

Proposed method

• Equip a frozen CLIP model with a lightweight, single-layer fusion transformer to refine image and text embeddings using retrieved knowledge.
• Use uni-modal retrieval: query image representations to retrieve top-k similar images, and query text representations to retrieve top-k similar texts.
• Fuse the retrieved items (images or texts) with the original query to form enriched multi-modal representations.
• Train the fusion module in a contrastive learning setup using a curated subset of WebLI (61M examples) to align the enhanced representations.
• Leverage approximate k-NN search for efficient inference on large-scale memory (e.g., WebLI with 1B entries).
• Ensure the method remains general-purpose by not requiring task-specific adaptation or retraining after deployment.

Experimental results

Research questions

• RQ1 Can external knowledge retrieval improve zero-shot performance on fine-grained vision-language tasks beyond standard contrastive pre-training?
• RQ2 Does uni-modal retrieval (image-to-image or text-to-text) outperform cross-modal retrieval for knowledge retrieval in this context?
• RQ3 How does the number of retrieved items affect performance and inference efficiency?
• RQ4 Can the model generalize to updated or larger memories without retraining?

Key findings

• RECO achieves a +10.9 absolute improvement on Stanford Cars, +10.2 on CUB-2011, and +7.3 on the OVEN benchmark over CLIP.
• Using only one retrieved item already improves performance over CLIP, with gains saturating at k=10 retrievals.
• The model generalizes to larger, unseen memories: when evaluated on full WebLI after training on only 1% of it, performance improves significantly.
• Training a separate MLP head on top of CLIP with the same data fails to improve performance, validating the superiority of retrieval-augmented learning.
• Uni-modal retrieval (image-to-image or text-to-text) produces more relevant matches than cross-modal retrieval, leading to better fused representations.
• The method maintains strong performance even with reduced retrieval counts, enabling efficient inference with minimal computational overhead.