[Paper Review] Contrastive Quantization with Code Memory for Unsupervised Image Retrieval

Jinpeng Wang, Ziyun Zeng+3 more|arXiv (Cornell University)|Sep 11, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

MeCoQ proposes a novel unsupervised deep quantization method that leverages contrastive learning with a novel quantization code memory to improve image retrieval without labeled data. By introducing codeword diversity regularization and a low-drift code memory, MeCoQ achieves state-of-the-art performance on benchmark datasets, outperforming prior methods by up to 2.77 average MAP points.

ABSTRACT

The high efficiency in computation and storage makes hashing (including binary hashing and quantization) a common strategy in large-scale retrieval systems. To alleviate the reliance on expensive annotations, unsupervised deep hashing becomes an important research problem. This paper provides a novel solution to unsupervised deep quantization, namely Contrastive Quantization with Code Memory (MeCoQ). Different from existing reconstruction-based strategies, we learn unsupervised binary descriptors by contrastive learning, which can better capture discriminative visual semantics. Besides, we uncover that codeword diversity regularization is critical to prevent contrastive learning-based quantization from model degeneration. Moreover, we introduce a novel quantization code memory module that boosts contrastive learning with lower feature drift than conventional feature memories. Extensive experiments on benchmark datasets show that MeCoQ outperforms state-of-the-art methods. Code and configurations are publicly available at https://github.com/gimpong/AAAI22-MeCoQ.

Motivation & Objective

• To address the limitations of reconstruction-based unsupervised deep hashing, which rely heavily on pre-trained backbones and suffer from poor generalization in target domains.
• To overcome challenges in contrastive learning-based quantization, including sampling bias, model degeneration, and inefficiency in small-batch training.
• To improve the effectiveness and efficiency of memory-based contrastive learning in unsupervised quantization by designing a low-drift code memory module.
• To demonstrate that codeword diversity regularization is essential to prevent model degeneration in contrastive quantization.

Proposed method

• MeCoQ employs a debiased contrastive learning framework to correct sampling bias in small-batch training without requiring large-scale data or complex synchronization.
• It introduces a codeword diversity regularization term that explicitly encourages diversity among quantization codewords to prevent representation collapse.
• A novel quantization code memory is designed to store and reuse quantized codes as negative samples, reducing feature drift compared to traditional feature memories.
• The code memory enables stable negative sampling over time, allowing effective contrastive learning even with limited batch sizes and without a momentum encoder.
• The model jointly optimizes contrastive loss and quantization loss, with the code memory providing persistent negative pairs to enhance representation learning.

Experimental results

Research questions

• RQ1 Can contrastive learning be effectively applied to unsupervised deep quantization without relying on expensive annotations?
• RQ2 How can model degeneration be prevented in contrastive quantization, particularly due to codeword collapse?
• RQ3 Can a memory mechanism for quantization codes outperform feature memory in terms of stability and performance?
• RQ4 Does delaying the use of the code memory improve training stability and final performance?

Key findings

• MeCoQ outperforms MeCoQ without codeword diversity regularization by 25.11, 30.34, 29.14, and 25.32 average MAP points on four benchmark datasets, proving the effectiveness of the regularization.
• MeCoQ achieves a 2.22–2.77 average MAP improvement over the variant without the code memory, demonstrating the value of the memory module.
• The soft code memory reduces feature drift and improves performance over both feature memory and hard code memory, with MeCoQ outperforming the feature memory variant by 0.44 to 2.52 average MAP points.
• MeCoQ with delayed memory usage improves performance by 0.60 to 3.36 average MAP points, indicating that early memory use leads to suboptimal representations.
• Using a memory size of 896 achieves the best balance, with further increases yielding diminishing returns due to feature drift.
• MeCoQ achieves better performance with significantly less GPU memory (5.9 GB) than increasing batch size to 256 (10.9 GB), proving its efficiency.