[Paper Review] Ranking-based Deep Cross-modal Hashing

Xuanwu Liu, Guoxian Yu+4 more|arXiv (Cornell University)|May 11, 2019

Advanced Image and Video Retrieval Techniques24 references4 citations

TL;DR

This paper proposes Ranking-based Deep Cross-modal Hashing (RDCMH), a novel method that integrates semi-supervised semantic ranking with deep feature learning to improve cross-modal retrieval. By jointly optimizing deep features and hash functions using adaptive triplet ranking loss, RDCMH achieves state-of-the-art performance on multi-modal datasets, especially under weakly supervised settings with limited labels.

ABSTRACT

Cross-modal hashing has been receiving increasing interests for its low storage cost and fast query speed in multi-modal data retrievals. However, most existing hashing methods are based on hand-crafted or raw level features of objects, which may not be optimally compatible with the coding process. Besides, these hashing methods are mainly designed to handle simple pairwise similarity. The complex multilevel ranking semantic structure of instances associated with multiple labels has not been well explored yet. In this paper, we propose a ranking-based deep cross-modal hashing approach (RDCMH). RDCMH firstly uses the feature and label information of data to derive a semi-supervised semantic ranking list. Next, to expand the semantic representation power of hand-crafted features, RDCMH integrates the semantic ranking information into deep cross-modal hashing and jointly optimizes the compatible parameters of deep feature representations and of hashing functions. Experiments on real multi-modal datasets show that RDCMH outperforms other competitive baselines and achieves the state-of-the-art performance in cross-modal retrieval applications.

Motivation & Objective

• To address the limitation of existing cross-modal hashing methods that rely on pairwise similarity and raw/hand-crafted features.
• To exploit complex, multi-level ranking semantics from multi-label data for improved retrieval.
• To enhance hashing performance by integrating both labeled and unlabeled data through a semi-supervised semantic measure.
• To jointly optimize deep feature learning and hash function learning for better compatibility and representation.
• To reduce dependency on complete label sets, making the method robust in weakly supervised scenarios.

Proposed method

• RDCMH constructs a semi-supervised semantic ranking list using both feature and label information to capture multi-level similarity.
• It introduces an adaptive triplet ranking loss that assigns higher weights to more relevant instance pairs based on semantic similarity.
• The method jointly optimizes deep neural network parameters for feature extraction and hashing functions using a unified objective.
• A label and feature-informed similarity matrix S is used to guide the ranking list construction, improving semantic representation.
• The framework integrates deep feature learning with hashing optimization in an end-to-end manner, enhancing compatibility between features and binary codes.
• It uses a scalar hyperparameter λ to balance the quantization loss and the ranking loss during training.

Experimental results

Research questions

• RQ1 Can a ranking-based approach that captures multi-level semantic structure improve cross-modal hashing beyond pairwise similarity?
• RQ2 How can unlabeled data be effectively leveraged to enhance hashing performance when labels are incomplete?
• RQ3 Does joint optimization of deep features and hash codes lead to better retrieval accuracy than separate learning?
• RQ4 How sensitive is the proposed method to hyperparameter choices, particularly λ?
• RQ5 Can the method maintain strong performance under weakly supervised settings with missing labels?

Key findings

• RDCMH achieves state-of-the-art performance on three benchmark multi-modal datasets (Mirflickr, NUS-WIDE, Wiki), outperforming all competitive baselines.
• On the Wiki dataset, RDCMH achieves higher MAP values than SePH, demonstrating superior performance in cross-modal retrieval.
• When 30% of labels were masked, RDCMH maintained a 4% higher average MAP than the second-best method (CDQ), showing robustness to label scarcity.
• Under 10% label availability, RDCMH’s MAP dropped by only 70.2%, compared to 81.9% for the second-best method (CHN), confirming its effectiveness in weakly supervised scenarios.
• Ablation studies show that RDCMH-NW (with adaptive weights) outperforms RDCMH-NW (uniform weights), proving the benefit of weighted triplet ranking.
• RDCMH significantly outperforms variants without deep feature learning (RDCMH-ND) and without joint optimization (RDCMH-NJ), confirming the advantage of end-to-end joint learning.