[Paper Review] Vision-by-Language for Training-Free Compositional Image Retrieval

Shyamgopal Karthik, Karsten Roth+2 more|arXiv (Cornell University)|Oct 13, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes CIReVL, a training-free approach for zero-shot compositional image retrieval that leverages vision-language models for image captioning and large language models for reasoning over textual modifications. By decoupling image understanding, language-based reasoning, and cross-modal retrieval, CIReVL achieves state-of-the-art performance—surpassing prior methods by up to 2.5× on mAP@5 in CIRCO and reaching R@1=34.64% on CIRR—while enabling human interpretability and post-hoc intervention through modular, language-based pipeline components.

ABSTRACT

Given an image and a target modification (e.g an image of the Eiffel tower and the text "without people and at night-time"), Compositional Image Retrieval (CIR) aims to retrieve the relevant target image in a database. While supervised approaches rely on annotating triplets that is costly (i.e. query image, textual modification, and target image), recent research sidesteps this need by using large-scale vision-language models (VLMs), performing Zero-Shot CIR (ZS-CIR). However, state-of-the-art approaches in ZS-CIR still require training task-specific, customized models over large amounts of image-text pairs. In this work, we propose to tackle CIR in a training-free manner via our Compositional Image Retrieval through Vision-by-Language (CIReVL), a simple, yet human-understandable and scalable pipeline that effectively recombines large-scale VLMs with large language models (LLMs). By captioning the reference image using a pre-trained generative VLM and asking a LLM to recompose the caption based on the textual target modification for subsequent retrieval via e.g. CLIP, we achieve modular language reasoning. In four ZS-CIR benchmarks, we find competitive, in-part state-of-the-art performance - improving over supervised methods. Moreover, the modularity of CIReVL offers simple scalability without re-training, allowing us to both investigate scaling laws and bottlenecks for ZS-CIR while easily scaling up to in parts more than double of previously reported results. Finally, we show that CIReVL makes CIR human-understandable by composing image and text in a modular fashion in the language domain, thereby making it intervenable, allowing to post-hoc re-align failure cases. Code will be released upon acceptance.

Motivation & Objective

• To address the high annotation cost of supervised compositional image retrieval (CIR) by enabling zero-shot retrieval without training.
• To improve interpretability and user intervention in CIR by operating primarily in the language domain.
• To explore scaling laws in zero-shot CIR without retraining, leveraging modular, plug-and-play components.
• To demonstrate that off-the-shelf models can achieve competitive or superior performance compared to task-specific, fine-tuned models.
• To enable post-hoc correction of retrieval failures through human intervention at any pipeline stage.

Proposed method

• Use a pre-trained vision-language model (e.g., BLIP-2 or CoCa) to generate a detailed caption of the reference image.
• Feed the generated caption and the textual modification (e.g., 'without people and at night-time') to a large language model (LLM) to generate a target caption describing the desired modified image.
• Use a pre-trained CLIP model to retrieve the most similar image in the database based on the LLM-generated caption.
• Decouple the retrieval pipeline into three modular stages: captioning, language-based reasoning, and cross-modal retrieval.
• Enable user intervention by allowing manual correction of captions or instructions at any stage without retraining.
• Scale performance by simply replacing the retrieval model (e.g., from CLIP-B/16 to ViT-g/14) without retraining the entire system.

Experimental results

Research questions

• RQ1 Can a training-free, modular pipeline using off-the-shelf models achieve competitive or superior performance in zero-shot compositional image retrieval compared to fine-tuned methods?
• RQ2 To what extent does language-level reasoning in the LLM stage improve retrieval accuracy and generalization?
• RQ3 How do scaling laws in retrieval model capacity affect performance in zero-shot CIR without retraining?
• RQ4 Can the modular, language-based design enable human interpretability and post-hoc intervention to correct retrieval failures?
• RQ5 Does the method generalize to diverse CIR benchmarks, including domain conversion and conditional similarity tasks?

Key findings

• CIReVL achieves R@1=34.64% on the CIRR benchmark, significantly outperforming the previous SOTA score of 24.55% reported by SEARLE.
• On the CIRCO benchmark, CIReVL reaches mAP@5=26.77%, more than doubling the highest previously reported score of 11.68%.
• The method demonstrates a clear log-linear scaling relationship between retrieval model capacity and performance, with larger models yielding substantial gains.
• Failure cases are easily diagnosable and correctable due to the language-based pipeline, enabling effective human intervention at the captioning or instruction level.
• Ablation studies confirm that language-level reasoning over the textual query is critical, while the captioning and retrieval components also contribute significantly to overall performance.
• The modular, training-free design allows seamless scaling and replacement of components, such as upgrading from CLIP-B/16 to ViT-g/14, without retraining or architectural changes.