[Paper Review] Approximate Nearest Neighbor Search in $\ell_p$

Huy L. Nguyên|arXiv (Cornell University)|Jun 15, 2013

Advanced Image and Video Retrieval Techniques8 references3 citations

TL;DR

This paper presents a new locality-sensitive hashing (LSH) scheme for approximate nearest neighbor search in ℓₚ space with 1 < p < 2, achieving a query time of O(dn^ρ) and space O(dn + n^{1+ρ}) with ρ = O((ln c)² / c^p), improving upon the prior ρ ≤ 1/c bound and approaching the theoretical lower bound ρ ≥ 1/c^p.

ABSTRACT

We present a new locality sensitive hashing (LSH) algorithm for $c$-approximate nearest neighbor search in $\ell_p$ with $1

Motivation & Objective

• Address the gap between known upper bounds and the theoretical lower bound for ρ in ℓₚ approximate nearest neighbor search with 1 < p < 2.
• Develop a simple and generalizable LSH framework that improves upon the prior ρ ≤ 1/c bound by Datar et al. (2004).
• Achieve ρ close to the information-theoretic lower bound ρ ≥ 1/c^p established by O’Donnell, Wu, and Zhou (2011).
• Generalize the ℓ₂ LSH approach of Andoni and Indyk (2006) to ℓₚ with 1 < p < 2 using p-stable distributions and lattice-based partitioning.

Proposed method

• Use p-stable distributions to project d-dimensional ℓₚ points into a t-dimensional space with t = Θ((c ln c)^p), preserving relative distances in expectation.
• Apply a random shift to a lattice Lₜ with spacing Δw = 4w, where w = O(c ln c), to create shifted lattices that cover the t-dimensional space.
• Define the hash function as the pair (u, y), where u identifies the shifted lattice and y is the closest lattice point such that the projected point lies within a ℓₚ ball of radius w.
• Leverage concentration bounds for p-stable distributions to ensure that distances are preserved with high probability during dimensionality reduction.
• Bound the probability of collision for close points (‖x−y‖ₚ ≤ 1) and distant points (‖x−y‖ₚ ≥ c) using volume arguments in ℓₚ balls.
• Derive ρ via the ratio ln(1/p₁)/ln(1/p₂), where p₁ and p₂ are collision probabilities for close and far pairs, respectively.

Experimental results

Research questions

• RQ1 Can the ρ parameter in ℓₚ approximate nearest neighbor search be improved beyond the prior upper bound of ρ ≤ 1/c for 1 < p < 2?
• RQ2 To what extent can the ℓ₂ LSH framework of Andoni and Indyk (2006) be generalized to ℓₚ with 1 < p < 2?
• RQ3 Is it possible to achieve ρ close to the information-theoretic lower bound ρ ≥ 1/c^p using a simple and analyzable LSH construction?
• RQ4 How do volume estimates of ℓₚ balls and their intersections under lattice partitions affect the collision probability in high-dimensional ℓₚ spaces?

Key findings

• The proposed LSH achieves ρ = O((ln c)² / c^p) for ℓₚ with 1 < p < 2, improving upon the prior upper bound of ρ ≤ 1/c.
• The bound ρ = O((ln c)² / c^p) is asymptotically close to the theoretical lower bound ρ ≥ 1/c^p, narrowing the gap between known upper and lower bounds.
• The construction uses p-stable projections followed by lattice-based partitioning, with the number of lattices Uₜ = Δ^t t^{t/p + 1} ln(Δt/δ) ensuring full space coverage with high probability.
• The analysis shows that with high probability, the projected distance between close points (‖x−y‖ₚ ≤ 1) is preserved within a factor O(ε⁻¹), while distant pairs (‖x−y‖ₚ ≥ c) are separated with high probability.
• The collision probability for close pairs is bounded via volume comparison of ℓₚ balls, yielding ln(1/p₁) ≤ O((2+γ) t / (p (2εw)^p)) for small γ.
• The collision probability for distant pairs is bounded using the uniform convexity and smoothness of ℓₚ norms, leading to ln(1/p₂) ≥ (1−o(1)) (p−1) t c² / (8w²), which dominates the ρ ratio.