[Paper Review] Consensus-Aware Visual-Semantic Embedding for Image-Text Matching

Haoran Wang, Ying Zhang+3 more|arXiv (Cornell University)|Jul 17, 2020

Multimodal Machine Learning Applications48 references17 citations

TL;DR

This paper proposes Consensus-Aware Visual-Semantic Embedding (CVSE), a novel framework that enhances image-text matching by integrating commonsense knowledge through statistical co-occurrence of semantic concepts from large-scale captioning corpora. By constructing a concept correlation graph to learn consensus-aware concept (CAC) representations, CVSE unifies instance-level and consensus-level features, achieving state-of-the-art performance on MS-COCO and Flickr30k for bidirectional retrieval.

ABSTRACT

Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations, thereby exploiting their matching relationships and making the corresponding alignments. Such approaches only exploit the superficial associations contained in the instance pairwise data, with no consideration of any external commonsense knowledge, which may hinder their capabilities to reason the higher-level relationships between image and text. In this paper, we propose a Consensus-aware Visual-Semantic Embedding (CVSE) model to incorporate the consensus information, namely the commonsense knowledge shared between both modalities, into image-text matching. Specifically, the consensus information is exploited by computing the statistical co-occurrence correlations between the semantic concepts from the image captioning corpus and deploying the constructed concept correlation graph to yield the consensus-aware concept (CAC) representations. Afterwards, CVSE learns the associations and alignments between image and text based on the exploited consensus as well as the instance-level representations for both modalities. Extensive experiments conducted on two public datasets verify that the exploited consensus makes significant contributions to constructing more meaningful visual-semantic embeddings, with the superior performances over the state-of-the-art approaches on the bidirectional image and text retrieval task. Our code of this paper is available at: https://github.com/BruceW91/CVSE.

Motivation & Objective

• To address the limitation of existing image-text matching methods that rely solely on instance-level alignment without leveraging external commonsense knowledge.
• To model high-level semantic relationships between images and texts by exploiting shared semantic concepts and their associations from large-scale captioning data.
• To unify instance-level representations with consensus-level representations to improve visual-semantic embedding quality and retrieval performance.
• To verify the generalization capability of consensus knowledge across different datasets, such as from MS-COCO to Flickr30k.

Proposed method

• Mining semantic concepts from image captioning corpora and computing their statistical co-occurrence frequencies to model consensus knowledge.
• Constructing a concept correlation graph from co-occurrence statistics to learn consensus-aware concept (CAC) representations using graph convolutional networks.
• Integrating CAC representations with instance-level image and text features via a learnable fusion module controlled by a weighting hyperparameter β.
• Training the model end-to-end using contrastive loss to align image-text pairs in the unified embedding space while preserving both instance-level and consensus-level semantics.
• Using t-SNE visualization to analyze the distribution and interpretability of CAC representations compared to word2vec embeddings.

Experimental results

Research questions

• RQ1 Can consensus knowledge derived from large-scale image captions improve visual-semantic embedding for image-text matching?
• RQ2 How does the integration of consensus-level representations with instance-level features affect retrieval performance?
• RQ3 To what extent can consensus knowledge generalize across different image-text datasets, such as from MS-COCO to Flickr30k?
• RQ4 Are the predicted consensus-aware concept representations semantically meaningful and aligned with human common sense?

Key findings

• CVSE achieves state-of-the-art performance on both MS-COCO and Flickr30k, with 56.4% R@1 and 89.0% R@10 for text retrieval, and 39.9% R@1 and 77.2% R@10 for image retrieval.
• The ablation study shows that using only instance-level features (CVSE β=1) or only consensus-level features (CVSE β=0) yields inferior performance, confirming the complementary nature of both representations.
• On cross-dataset generalization from MS-COCO to Flickr30k, CVSE outperforms all baselines, including re-implemented SOTA methods, demonstrating the robustness and transferability of consensus knowledge.
• Visualization results confirm that CAC representations capture meaningful semantic associations—e.g., 'riding' is closely linked to 'person', and 'plate' to 'eating'—validating their alignment with human common sense.
• The model successfully predicts relevant concepts not explicitly present in the input image-text pair, such as 'traffic' and 'buildings' for a surfing scene, enhancing interpretability and semantic richness.