[Paper Review] Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network

Yuxin Peng, Jinwei Qi+1 more|arXiv (Cornell University)|Aug 16, 2017

Multimodal Machine Learning Applications42 references3 citations

TL;DR

This paper proposes a modality-specific cross-modal similarity measurement (MCSM) approach using recurrent attention networks to model modality-specific semantics independently, enabling end-to-end learning of cross-modal similarity without explicit common representation. By leveraging modality-specific semantic spaces and adaptive fusion, the method outperforms 9 state-of-the-art methods on Wikipedia, Pascal Sentence, and the authors' XMediaNet datasets, demonstrating superior performance in capturing imbalanced and complementary relationships between modalities.

ABSTRACT

Nowadays, cross-modal retrieval plays an indispensable role to flexibly find information across different modalities of data. Effectively measuring the similarity between different modalities of data is the key of cross-modal retrieval. Different modalities such as image and text have imbalanced and complementary relationships, which contain unequal amount of information when describing the same semantics. For example, images often contain more details that cannot be demonstrated by textual descriptions and vice versa. Existing works based on Deep Neural Network (DNN) mostly construct one common space for different modalities to find the latent alignments between them, which lose their exclusive modality-specific characteristics. Different from the existing works, we propose modality-specific cross-modal similarity measurement (MCSM) approach by constructing independent semantic space for each modality, which adopts end-to-end framework to directly generate modality-specific cross-modal similarity without explicit common representation. For each semantic space, modality-specific characteristics within one modality are fully exploited by recurrent attention network, while the data of another modality is projected into this space with attention based joint embedding to utilize the learned attention weights for guiding the fine-grained cross-modal correlation learning, which can capture the imbalanced and complementary relationships between different modalities. Finally, the complementarity between the semantic spaces for different modalities is explored by adaptive fusion of the modality-specific cross-modal similarities to perform cross-modal retrieval. Experiments on the widely-used Wikipedia and Pascal Sentence datasets as well as our constructed large-scale XMediaNet dataset verify the effectiveness of our proposed approach, outperforming 9 state-of-the-art methods.

Motivation & Objective

• To address the limitation of existing cross-modal retrieval methods that treat different modalities equally by projecting them into a single common space, which loses modality-specific characteristics.
• To model the imbalanced and complementary nature of different modalities—such as images containing details not described in text and vice versa—by constructing independent semantic spaces for each modality.
• To enable end-to-end learning of cross-modal similarity without explicit common representation, directly generating modality-specific similarities through a unified framework.
• To exploit the complementarity between modality-specific semantic spaces via adaptive fusion to improve retrieval accuracy.

Proposed method

• Constructs independent semantic spaces for each modality (e.g., image and text) to preserve modality-specific characteristics during cross-modal correlation learning.
• Employs a recurrent attention network with attention-based joint embedding loss to model intra-modality dependencies and align cross-modal representations using attention weights.
• Projects data from one modality into the semantic space of another using attention-based joint embedding, enabling fine-grained correlation learning while preserving modality-specific features.
• Uses an end-to-end framework to directly generate modality-specific cross-modal similarities without requiring explicit common representation.
• Applies adaptive fusion to combine similarities from both semantic spaces, dynamically weighting contributions based on modality-specific relevance.
• Optimizes the entire network using a joint loss function that integrates both modality-specific representation learning and cross-modal similarity learning.

Experimental results

Research questions

• RQ1 How can cross-modal similarity be measured effectively when different modalities exhibit imbalanced and complementary information, such as images containing details not described in text?
• RQ2 Can independent semantic spaces for each modality preserve modality-specific characteristics better than a shared common space?
• RQ3 Does adaptive fusion of modality-specific similarities improve cross-modal retrieval performance compared to simple averaging or late fusion?
• RQ4 Can the proposed method outperform state-of-the-art approaches on diverse cross-modal retrieval benchmarks?

Key findings

• On the Wikipedia dataset, the proposed MCSM approach achieves a mean average precision (mAP) of 0.598, outperforming the next-best method by a significant margin.
• On the Pascal Sentence dataset, the MCSM approach achieves mAP of 0.598, surpassing the second-best method by 0.019 in mAP.
• On the large-scale XMediaNet dataset, the MCSM approach achieves mAP of 0.545, demonstrating strong generalization and scalability.
• Ablation studies show that MCSM-image outperforms MCSM-text on datasets with object-level semantics (e.g., Pascal Sentence and XMediaNet), while MCSM-text performs better on high-level semantic datasets (e.g., Wikipedia), confirming the imbalanced nature of modality information.
• Adaptive fusion consistently improves performance over late fusion (MCSM-LF), proving that the method effectively captures complementarity between semantic spaces.
• The proposed method outperforms 9 state-of-the-art methods across all three datasets, validating its effectiveness and robustness in cross-modal retrieval.