[Paper Review] Cosine Similarity Search with Multi Index Hashing.

Sepehr Eghbali, Ladan Tahvildari|arXiv (Cornell University)|Sep 14, 2016

Advanced Image and Video Retrieval Techniques42 references4 citations

TL;DR

This paper proposes a multi-index hashing method that enables fast, exact cosine similarity search in large-scale binary-coded datasets by exploiting the mathematical relationship between Hamming distance and cosine similarity. By deriving a sequence of hash table buckets to probe, the approach achieves order-of-magnitude speedups over exhaustive search and approximate methods like LSH, while maintaining exact retrieval of the K most similar items.

ABSTRACT

Due to rapid development of the Internet, recent years have witnessed an explosion in the rate of data generation. Dealing with data at current scales brings up unprecedented challenges. From the algorithmic view point, executing existing linear algorithms in information retrieval and machine learning on such tremendous amounts of data incur intolerable computational and storage costs. To address this issue, there is a growing interest to map data points in large-scale datasets to binary codes. This can significantly reduce the storage complexity of large-scale datasets. However, one of the most compelling reasons for using binary codes or any discrete representation is that they can be used as direct indices into a hash table. Incorporating hash table offers fast query execution; one can look up the nearby buckets in a hash table populated with binary codes to retrieve similar items. Nonetheless, if binary codes are compared in terms of the cosine similarity rather than the Hamming distance, there is no fast exact sequential procedure to find the $K$ closest items to the query other than the exhaustive search. Given a large dataset of binary codes and a binary query, the problem that we address is to efficiently find $K$ closest codes in the dataset that yield the largest cosine similarities to the query. To handle this issue, we first elaborate on the relation between the Hamming distance and the cosine similarity. This allows finding the sequence of buckets to check in the hash table. Having this sequence, we propose a multi-index hashing approach that can increase the search speed up to orders of magnitude in comparison to the exhaustive search and even approximation methods such as LSH. We empirically evaluate the performance of the proposed algorithm on real world datasets.

Motivation & Objective

• Address the challenge of efficiently finding the K most similar items to a query in large-scale datasets represented as binary codes using cosine similarity.
• Overcome the limitation that cosine similarity cannot be efficiently computed via standard hash table lookups, unlike Hamming distance.
• Develop a method that enables fast, exact retrieval of K nearest neighbors under cosine similarity using binary codes.
• Reduce computational cost and storage complexity in large-scale information retrieval and machine learning systems.
• Achieve significant speedup over exhaustive search and existing approximate methods like LSH while maintaining exactness.

Proposed method

• Establish a theoretical relationship between Hamming distance and cosine similarity to guide efficient bucket traversal in hash tables.
• Design a multi-index hashing structure that maps binary codes to multiple hash tables for improved search coverage and efficiency.
• Derive a sequence of hash table buckets to probe based on the cosine similarity-Hamming distance duality, enabling fast lookup without exhaustive comparison.
• Use the derived bucket traversal order to systematically search for the K closest binary codes to the query in terms of cosine similarity.
• Leverage the structure of binary codes and their geometric properties to avoid full dataset scans during similarity search.
• Implement a query processing pipeline that uses the precomputed bucket sequence to retrieve candidates with minimal computation.

Experimental results

Research questions

• RQ1 Can the relationship between Hamming distance and cosine similarity be exploited to design a fast, exact search procedure for binary codes?
• RQ2 Is it possible to achieve exact K-nearest neighbor retrieval under cosine similarity using only hash table lookups, without exhaustive search?
• RQ3 How does the proposed multi-index hashing method compare in speed and accuracy to exhaustive search and LSH-based approaches?
• RQ4 Can the proposed method scale efficiently to real-world large-scale datasets while maintaining exactness?
• RQ5 What is the optimal sequence of hash table buckets to probe for maximizing retrieval speed and accuracy?

Key findings

• The proposed method achieves order-of-magnitude speedups over exhaustive search in cosine similarity search on large-scale binary-coded datasets.
• The method enables exact retrieval of the K most similar items under cosine similarity, unlike approximate methods such as LSH.
• By leveraging the duality between Hamming distance and cosine similarity, the method reduces the number of hash table probes required per query.
• Empirical evaluation on real-world datasets confirms that the proposed approach outperforms both exhaustive search and LSH in terms of query speed.
• The multi-index hashing structure improves search efficiency by increasing the likelihood of retrieving high-similarity items through multiple independent hash tables.
• The method maintains high retrieval accuracy while drastically reducing computational cost, making it suitable for large-scale applications.