[Paper Review] Cross Modal Retrieval with Querybank Normalisation

Simion-Vlad Bogolin, Ioana Croitoru+3 more|arXiv (Cornell University)|Dec 23, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Querybank Normalisation (QB-Norm), a non-parametric framework that mitigates the hubness problem in cross-modal retrieval by re-normalising query-gallery similarities using a querybank of gallery embeddings. It introduces Dynamic Inverted Softmax (DIS) for robust similarity normalisation and achieves state-of-the-art performance across multiple models and benchmarks without retraining or requiring concurrent test queries.

ABSTRACT

Profiting from large-scale training datasets, advances in neural architecture design and efficient inference, joint embeddings have become the dominant approach for tackling cross-modal retrieval. In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries. Drawing inspiration from the NLP literature, we formulate a simple but effective framework called Querybank Normalisation (QB-Norm) that re-normalises query similarities to account for hubs in the embedding space. QB-Norm improves retrieval performance without requiring retraining. Differently from prior work, we show that QB-Norm works effectively without concurrent access to any test set queries. Within the QB-Norm framework, we also propose a novel similarity normalisation method, the Dynamic Inverted Softmax, that is significantly more robust than existing approaches. We showcase QB-Norm across a range of cross modal retrieval models and benchmarks where it consistently enhances strong baselines beyond the state of the art. Code is available at https://vladbogo.github.io/QB-Norm/.

Motivation & Objective

• To address the persistent hubness problem in high-dimensional joint embeddings used for cross-modal retrieval.
• To develop a retrieval framework that improves performance without requiring model fine-tuning or retraining.
• To enable effective hubness mitigation in inference without concurrent access to multiple test queries.
• To design a robust similarity normalisation method that is insensitive to querybank selection.
• To demonstrate generalisation across diverse models, modalities, and benchmarks.

Proposed method

• QB-Norm re-normalises similarity scores between queries and gallery samples using a querybank of gallery embeddings to reduce the influence of hub nodes.
• The querybank is constructed from gallery embeddings during inference, enabling hubness correction without retraining.
• Dynamic Inverted Softmax (DIS) is introduced as a novel normalisation mechanism that adaptively adjusts similarity scores based on hub frequency.
• DIS uses an inverse temperature parameter to control the degree of softening, with optimal performance observed at β=20.
• The framework is applied post-inference, making it compatible with any pre-trained cross-modal embedding model.
• QB-Norm operates solely on similarity scores, requiring no architectural changes or additional training.

Figure 1 : Left: The hubness problem. We consider the problem of cross modal retrieval in which queries $q_{1}$ and $q_{2}$ are compared against a gallery of samples, $x_{1}$ and $x_{2}$ . As we show in Sec. 3.2 , the high-dimensional joint embeddings employed by modern methods for cross-modal retri

Experimental results

Research questions

• RQ1 Can hubness be effectively mitigated in modern cross-modal retrieval systems without retraining?
• RQ2 Does QB-Norm remain effective when only a single test query is available at inference time?
• RQ3 Can a similarity normalisation method be designed that is robust to querybank selection and improves retrieval performance across diverse models?
• RQ4 How does QB-Norm compare to existing hubness reduction techniques in terms of performance and robustness?
• RQ5 Does QB-Norm generalize across different modalities and retrieval benchmarks?

Key findings

• QB-Norm achieves a 5.9% absolute improvement in R@1 on the MSR-VTT v2t retrieval task, rising from 24.6% to 30.1% when applied to TT-CE+.
• On the same benchmark, R@5 increases from 54.1% to 61.4%, and R@10 from 67.5% to 73.2% with QB-Norm.
• The method consistently improves retrieval performance across text-to-video, text-to-image, and audio-to-text retrieval tasks.
• Performance saturates with larger querybanks, indicating diminishing returns beyond a certain size.
• Dynamic Inverted Softmax (DIS) outperforms prior normalisation methods and is robust to querybank selection, avoiding performance degradation on certain banks.
• QB-Norm maintains effectiveness even when only a single query is available at inference, unlike prior methods requiring multiple queries.

Figure 2 : Hubness is pervasive in leading methods for text-video retrieval . The charts depict the distribution of the number of times each gallery video was retrieved by test set queries (x-axes video ids are ordered by decreasing retrieval count). Top row (different models): We report retrieval d