[Paper Review] Revisiting Cross Modal Retrieval

Shah Nawaz, Muhammad Kamran Janjua+2 more|arXiv (Cornell University)|Jul 19, 2018

Multimodal Machine Learning Applications57 references4 citations

TL;DR

This paper proposes a single-network cross-modal retrieval system that fuses image and text encodings into a shared embedding space, eliminating the need for separate networks per modality. By leveraging InceptionResNet-V1 with center loss for joint training, it achieves competitive performance on MS-COCO and Flickr30K, demonstrating that semantic relationships between modalities can be effectively captured through unified text encoding, challenging the dominance of multi-network architectures.

ABSTRACT

This paper proposes a cross-modal retrieval system that leverages on image and text encoding. Most multimodal architectures employ separate networks for each modality to capture the semantic relationship between them. However, in our work image-text encoding can achieve comparable results in terms of cross-modal retrieval without having to use a separate network for each modality. We show that text encodings can capture semantic relationships between multiple modalities. In our knowledge, this work is the first of its kind in terms of employing a single network and fused image-text embedding for cross-modal retrieval. We evaluate our approach on two famous multimodal datasets: MS-COCO and Flickr30K.

Motivation & Objective

• To develop a unified deep learning architecture for cross-modal retrieval that avoids separate networks for image and text.
• To evaluate whether text encodings alone can capture semantic relationships across modalities, reducing reliance on paired image-text data.
• To challenge the validity of Recall@K as the primary evaluation metric for multimodal retrieval systems.
• To demonstrate that a single, scalable network can effectively map both image and text into a shared embedding space.

Proposed method

• Uses a single InceptionResNet-V1 backbone to process both image and text-encoded inputs in a shared network.
• Employs text encoding via pre-trained sentence embeddings to represent textual descriptions in the joint space.
• Applies center loss as a supervision signal to minimize intra-class distances between image-text pairs.
• Fuses image and text representations early in the network to enable joint learning of semantic relationships.
• Trains the model end-to-end using contrastive loss to encourage semantic similarity between matching image-text pairs.
• Performs retrieval via Euclidean distance in the shared embedding space, ranking nearest neighbors for evaluation.

Experimental results

Research questions

• RQ1 Can a single network effectively learn joint image-text representations without separate encoders for each modality?
• RQ2 To what extent can text encodings alone capture semantic relationships between images and their descriptions?
• RQ3 How does the performance of a unified network compare to multi-network state-of-the-art methods on standard benchmarks?
• RQ4 Why does Recall@K fail to reflect semantic similarity in retrieval results, and what alternative evaluation criteria are more appropriate?

Key findings

• The proposed single-network approach achieves competitive Recall@10 scores of 45.76 on MS-COCO and 42.6 on Flickr30K for sentence-to-image retrieval, despite using a unified architecture.
• The model achieves 30.10 Recall@5 on MS-COCO and 33.0 on Flickr30K for image-to-sentence retrieval, demonstrating robust cross-modal alignment.
• The authors argue that Recall@K is a poor metric for evaluating semantic retrieval, as it fails to account for semantically similar but non-matching pairs, as illustrated in Figure 1.
• Results show that even when retrieved results are semantically plausible, low Recall@K scores occur if the exact paired example is not retrieved, highlighting a key limitation of the metric.
• The study demonstrates that text encodings alone can effectively capture cross-modal semantic relationships, making separate modality-specific networks unnecessary.