[Paper Review] Locality-Sensitive Hashing for f-Divergences: Mutual Information Loss and Beyond

Lin Chen, Hossein Esfandiari+2 more|arXiv (Cornell University)|Oct 28, 2019

Face and Expression Recognition4 citations

TL;DR

This paper proposes locality-sensitive hashing (LSH) schemes for f-divergences and Kreĭn kernels, enabling efficient approximate nearest neighbor search in high-dimensional probability spaces. It introduces a general framework for f-divergences using Hellinger approximation with provable two-sided bounds, and a lossless reduction to maximum inner product search for mutual information loss, enabling scalable similarity search in applications like model compression and recommendation systems.

ABSTRACT

Computing approximate nearest neighbors in high dimensional spaces is a central problem in large-scale data mining with a wide range of applications in machine learning and data science. A popular and effective technique in computing nearest neighbors approximately is the locality-sensitive hashing (LSH) scheme. In this paper, we aim to develop LSH schemes for distance functions that measure the distance between two probability distributions, particularly for f-divergences as well as a generalization to capture mutual information loss. First, we provide a general framework to design LHS schemes for f-divergence distance functions and develop LSH schemes for the generalized Jensen-Shannon divergence and triangular discrimination in this framework. We show a two-sided approximation result for approximation of the generalized Jensen-Shannon divergence by the Hellinger distance, which may be of independent interest. Next, we show a general method of reducing the problem of designing an LSH scheme for a Krein kernel (which can be expressed as the difference of two positive definite kernels) to the problem of maximum inner product search. We exemplify this method by applying it to the mutual information loss, due to its several important applications such as model compression.

Motivation & Objective

• Develop a general LSH framework for f-divergences to enable scalable approximate nearest neighbor search in high-dimensional probability distributions.
• Address the lack of provable approximation guarantees in prior work on Jensen-Shannon divergence and related f-divergences.
• Provide a lossless reduction from Kreĭn kernel-based similarity (e.g., mutual information loss) to maximum inner product search (MIPS).
• Enable practical deployment of LSH for information-theoretic distances in real-world datasets such as MNIST, Fashion MNIST, and CIFAR-10.
• Demonstrate robust performance trade-offs between precision and speed-up in large-scale similarity search tasks.

Proposed method

• Propose a general reduction tool (Proposition 1) to design LSH families for f-divergences by leveraging the Hellinger distance as a proxy.
• Establish a two-sided approximation bound between the generalized Jensen-Shannon (GJS) divergence and the Hellinger distance, enabling theoretical guarantees.
• Apply the framework to construct LSH schemes for two specific f-divergences: generalized Jensen-Shannon divergence and triangular discrimination.
• Introduce a general reduction from Kreĭn kernel-based similarity (e.g., mutual information loss) to maximum inner product search (MIPS), enabling use of existing MIPS-optimized LSH families.
• Use random-projection-based LSH for cosine similarity to implement the MIPS component, with a compound hash function formed by concatenating multiple hash functions.
• Implement a norm-ranging partitioning strategy to mitigate performance degradation from the $ ext{sign}(fx^ op fa) $-based hash function in high-dimensional spaces.

Experimental results

Research questions

• RQ1 Can we design an LSH scheme for f-divergences with provable approximation guarantees, particularly for generalized Jensen-Shannon divergence?
• RQ2 How well can the Hellinger distance approximate the generalized Jensen-Shannon divergence, and what are the tightness bounds of this approximation?
• RQ3 Can we reduce the problem of LSH for Kreĭn kernels to maximum inner product search (MIPS) in a theoretically lossless manner?
• RQ4 How does the proposed LSH scheme for mutual information loss perform in terms of precision and speed-up in real-world image datasets?
• RQ5 What is the trade-off between computational efficiency (speed-up) and retrieval quality (precision) in large-scale approximate nearest neighbor search using the proposed method?

Key findings

• The generalized Jensen-Shannon divergence is tightly two-sidedly approximated by the Hellinger distance, with empirical validation showing both upper and lower bounds are tight across different λ values (1/2, 1/3, 1/10) on Reuters-21578 topic distributions.
• On Fashion MNIST, MNIST, and CIFAR-10, the LSH-based k-NN search achieves a speed-up factor of up to 10x with precision above 0.8 when using K=3 and L=40 hash functions.
• Increasing the number of concatenated hash functions K from 3 to 5 improves efficiency at the same target precision, demonstrating that higher K yields better performance for a fixed precision level.
• The method achieves robust performance across different λ values for the generalized Jensen-Shannon divergence, with no significant drop in precision, indicating stability to parameter choice.
• The LSH scheme for mutual information loss via the MIPS reduction is theoretically lossless and enables efficient similarity search in applications such as model compression and recommendation systems.
• The norm-ranging partitioning strategy effectively mitigates performance degradation from the $ ext{sign}(fx^ op fa) $-based hash function, especially in high-dimensional spaces where $ orm{fx}_2 $ is large.