[Paper Review] RACE: Sub-Linear Memory Sketches for Approximate Near-Neighbor Search on Streaming Data.

Benjamin Coleman, Anshumali Shrivastava+1 more|arXiv (Cornell University)|Feb 18, 2019

Sparse and Compressive Sensing Techniques20 references4 citations

TL;DR

RACE introduces the first sub-linear memory sketch for approximate near-neighbor search on streaming data, combining LSH-based estimation (ACE), compressed sensing, and heavy hitter techniques to compress N high-dimensional vectors into a sketch of size O(N^b log²N) with b < 1. This enables efficient, high-probability top-v near-neighbor retrieval while breaking the traditional O(N) memory barrier.

ABSTRACT

We demonstrate the first possibility of a sub-linear memory sketch for solving the approximate near-neighbor search problem. In particular, we develop an online sketching algorithm that can compress $N$ vectors into a tiny sketch consisting of small arrays of counters whose size scales as $O(N^{b}\log^2{N})$, where $b < 1$ depending on the stability of the near-neighbor search. This sketch is sufficient to identify the top-$v$ near-neighbors with high probability. To the best of our knowledge, this is the first near-neighbor search algorithm that breaks the linear memory ($O(N)$) barrier. We achieve sub-linear memory by combining advances in locality sensitive hashing (LSH) based estimation, especially the recently-published ACE algorithm, with compressed sensing and heavy hitter techniques. We provide strong theoretical guarantees; in particular, our analysis sheds new light on the memory-accuracy tradeoff in the near-neighbor search setting and the role of sparsity in compressed sensing, which could be of independent interest. We rigorously evaluate our framework, which we call RACE (Repeated ACE) data structures on a friend recommendation task on the Google plus graph with more than 100,000 high-dimensional vectors. RACE provides compression that is orders of magnitude better than the random projection based alternative, which is unsurprising given the theoretical advantage. We anticipate that RACE will enable both new theoretical perspectives on near-neighbor search and new methodologies for applications like high-speed data mining, internet-of-things (IoT), and beyond.

Motivation & Objective

• To overcome the O(N) memory bottleneck in approximate near-neighbor search for streaming data.
• To design a sketching framework that achieves sub-linear memory growth while preserving high accuracy in near-neighbor retrieval.
• To provide theoretical guarantees on memory-accuracy tradeoffs and the role of sparsity in compressed sensing for near-neighbor search.
• To enable practical deployment in memory-constrained environments such as IoT and high-speed data mining.

Proposed method

• RACE combines the ACE algorithm for LSH-based estimation with compressed sensing to enable efficient sketching of high-dimensional vectors.
• It employs heavy hitter techniques to identify and track the most significant components in the data stream, reducing sketch size.
• The sketch is constructed using small arrays of counters whose size scales as O(N^b log²N), with b < 1 depending on search stability.
• The framework supports online processing, allowing incremental updates to the sketch as new vectors arrive.
• It leverages sparsity in the data and compressed sensing reconstruction to maintain accuracy with minimal memory.
• Theoretical analysis establishes high-probability guarantees for top-v near-neighbor recovery.

Experimental results

Research questions

• RQ1 Can approximate near-neighbor search be achieved with sub-linear memory growth in the number of vectors?
• RQ2 How does sparsity in compressed sensing influence the memory-accuracy tradeoff in near-neighbor search?
• RQ3 What is the theoretical limit of sketch size for high-probability near-neighbor retrieval?
• RQ4 Can LSH-based estimation be effectively combined with compressed sensing and heavy hitters for streaming data?

Key findings

• RACE achieves a sketch size of O(N^b log²N) with b < 1, representing a strict sub-linear memory growth, breaking the O(N) memory barrier.
• The framework provides high-probability guarantees for identifying the top-v near-neighbors using the compact sketch.
• On the Google+ graph with over 100,000 high-dimensional vectors, RACE achieved compression orders of magnitude better than random projection.
• Theoretical analysis reveals new insights into the role of sparsity and memory-accuracy tradeoffs in compressed sensing for near-neighbor search.
• RACE enables practical deployment in memory-constrained applications such as IoT and real-time data mining.