[Paper Review] Efficient Cross-Modal Retrieval via Deep Binary Hashing and Quantization

Yang Shi, Youngjoo Chung|arXiv (Cornell University)|Feb 15, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes HQ, a deep learning framework that jointly learns binary hash codes and quantization codes for efficient cross-modal retrieval. By using a two-stage retrieval—first filtering candidates via Hamming distance on binary codes, then re-ranking with quantization distance—it achieves over 7% higher precision than state-of-the-art models while maintaining O(n) speedup compared to pure quantization methods.

ABSTRACT

Cross-modal retrieval aims to search for data with similar semantic meanings across different content modalities. However, cross-modal retrieval requires huge amounts of storage and retrieval time since it needs to process data in multiple modalities. Existing works focused on learning single-source compact features such as binary hash codes that preserve similarities between different modalities. In this work, we propose a jointly learned deep hashing and quantization network (HQ) for cross-modal retrieval. We simultaneously learn binary hash codes and quantization codes to preserve semantic information in multiple modalities by an end-to-end deep learning architecture. At the retrieval step, binary hashing is used to retrieve a subset of items from the search space, then quantization is used to re-rank the retrieved items. We theoretically and empirically show that this two-stage retrieval approach provides faster retrieval results while preserving accuracy. Experimental results on the NUS-WIDE, MIR-Flickr, and Amazon datasets demonstrate that HQ achieves boosts of more than 7% in precision compared to supervised neural network-based compact coding models.

Motivation & Objective

• Address the high storage and computational cost of cross-modal retrieval across multiple modalities such as images, text, and audio.
• Overcome the limitations of single-code methods—binary hashing (fast but less accurate) and quantization (accurate but slow)—by jointly learning both codes.
• Design a two-stage retrieval pipeline that leverages binary hashing for fast candidate filtering and quantization for accurate re-ranking.
• Achieve superior retrieval accuracy with reduced computational cost by end-to-end learning of both compact codes.
• Demonstrate that simultaneous learning of binary and quantization codes improves performance over sequential or regularized alternatives.

Proposed method

• Propose an end-to-end deep neural network that jointly optimizes binary hash codes and quantization codes for multi-modal data.
• Use Hamming distance between binary codes as the first-stage filter to drastically reduce the search space.
• Apply quantization distance (e.g., L1 or L2 norm between quantized vectors and codebook entries) in the second stage to re-rank the filtered candidates.
• Integrate both code types via a shared feature encoder and a shared loss function that preserves cross-modal semantic similarity.
• Leverage a learnable codebook for quantization, where each data point is assigned a binary code indicating its nearest dictionary entry.
• The model is trained end-to-end with a contrastive loss that encourages semantically similar samples to have similar codes across modalities.

Experimental results

Research questions

• RQ1 Can jointly learning binary hash codes and quantization codes improve cross-modal retrieval accuracy compared to using either code type alone?
• RQ2 Does a two-stage retrieval pipeline—using binary hashing for filtering and quantization for re-ranking—achieve better speed-accuracy trade-offs than single-stage methods?
• RQ3 How does simultaneous learning of both codes compare to training them sequentially or with regularization in isolation?
• RQ4 What is the optimal balance between the number of candidates retrieved via binary hashing and the accuracy of subsequent quantization-based re-ranking?
• RQ5 To what extent does the proposed HQ framework reduce retrieval time while preserving or improving accuracy compared to existing quantization-based models?

Key findings

• HQ achieves a precision boost of over 7% in mean average precision (MAP@50) compared to supervised neural network-based compact coding models on NUS-WIDE, MIR-Flickr, and Amazon datasets.
• The two-stage retrieval strategy with binary hashing followed by quantization re-ranking achieves higher accuracy than using only binary codes or only quantization codes.
• When α (the fraction of candidates retrieved via binary hashing) is set to 0.1–0.3 or 0.6–0.8, the model achieves the best trade-off between accuracy and speed.
• Retrieval time for HQ scales linearly with the input feature dimension n, achieving O(n) speedup over CDQ (quantization-only) methods while maintaining the same memory usage and accuracy.
• Jointly learning binary and quantization codes outperforms training them separately (e.g., C+D model), demonstrating that information is better shared across both code types.
• Quantization-based models with L1 or L2 regularization underperform HQ, confirming that joint optimization is more effective than regularization as a proxy for code interaction.