[Paper Review] Zero shot hashing

Shubham Pachori, Shanmuganathan Raman|arXiv (Cornell University)|Oct 9, 2016

Advanced Image and Video Retrieval Techniques42 references3 citations

TL;DR

This paper proposes a zero-shot hashing framework that generates hash codes for images of unseen object classes using only textual descriptions as auxiliary signals. By formulating the problem as unsupervised hashing, it achieves high-precision prediction of labels for unknown classes without requiring labeled examples during training.

ABSTRACT

This paper provides a framework to hash images containing instances of unknown object classes. In many object recognition problems, we might have access to huge amount of data. It may so happen that even this huge data doesn't cover the objects belonging to classes that we see in our day to day life. Zero shot learning exploits auxiliary information (also called as signatures) in order to predict the labels corresponding to unknown classes. In this work, we attempt to generate the hash codes for images belonging to unseen classes, information of which is available only through the textual corpus. We formulate this as an unsupervised hashing formulation as the exact labels are not available for the instances of unseen classes. We show that the proposed solution is able to generate hash codes which can predict labels corresponding to unseen classes with appreciably good precision.

Motivation & Objective

• To address the challenge of hashing images from unknown object classes not present in training data.
• To leverage textual descriptions as auxiliary signals for zero-shot label prediction in hashing.
• To develop an unsupervised hashing framework that generates compact binary codes for unseen classes.
• To enable accurate retrieval and classification of images belonging to novel object categories using only semantic information.

Proposed method

• Formulates zero-shot image hashing as an unsupervised learning problem due to the absence of labels for unseen classes.
• Uses textual embeddings as signatures to represent unseen classes and aligns them with image features in a shared embedding space.
• Designs a hashing loss function that preserves semantic similarity between image and text embeddings while enforcing binary code constraints.
• Optimizes the joint embedding and hashing function end-to-end using deep neural networks.
• Employs a contrastive learning strategy to pull together semantically similar image-text pairs and push apart dissimilar ones.
• Trains the model on seen classes only, enabling inference on unseen classes via textual descriptions alone.

Experimental results

Research questions

• RQ1 Can we generate effective hash codes for images of unknown object classes without any labeled examples during training?
• RQ2 How well can a hashing model trained on seen classes generalize to unseen classes using only textual descriptions?
• RQ3 To what extent does the use of textual signatures improve the precision of zero-shot label prediction in hashing?
• RQ4 Can an unsupervised hashing framework maintain semantic consistency between image and text representations for zero-shot generalization?

Key findings

• The proposed method achieves appreciably good precision in predicting labels for unseen object classes using only textual information.
• The framework successfully generates hash codes for images of unknown classes by leveraging semantic embeddings from text.
• The unsupervised nature of the approach enables effective zero-shot generalization without requiring labeled data for unseen classes.
• The model maintains semantic similarity between image and text representations, which is critical for accurate zero-shot prediction.
• The results demonstrate that textual signatures alone can guide the generation of meaningful hash codes for novel categories.