[Paper Review] SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search

Qi Chen, Bing Zhao+6 more|arXiv (Cornell University)|Nov 5, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

SPANN proposes a memory-disk hybrid approximate nearest neighbor search system that stores only centroid points in memory and large posting lists on disk, using hierarchical clustering and query-aware pruning to achieve low latency and high recall. It outperforms DiskANN by 2× in speed at 90% recall on billion-scale datasets with only 32GB memory, achieving sub-millisecond latency for recall@1 and recall@10.

ABSTRACT

The in-memory algorithms for approximate nearest neighbor search (ANNS) have achieved great success for fast high-recall search, but are extremely expensive when handling very large scale database. Thus, there is an increasing request for the hybrid ANNS solutions with small memory and inexpensive solid-state drive (SSD). In this paper, we present a simple but efficient memory-disk hybrid indexing and search system, named SPANN, that follows the inverted index methodology. It stores the centroid points of the posting lists in the memory and the large posting lists in the disk. We guarantee both disk-access efficiency (low latency) and high recall by effectively reducing the disk-access number and retrieving high-quality posting lists. In the index-building stage, we adopt a hierarchical balanced clustering algorithm to balance the length of posting lists and augment the posting list by adding the points in the closure of the corresponding clusters. In the search stage, we use a query-aware scheme to dynamically prune the access of unnecessary posting lists. Experiment results demonstrate that SPANN is 2$\ imes$ faster than the state-of-the-art ANNS solution DiskANN to reach the same recall quality $90\\%$ with same memory cost in three billion-scale datasets. It can reach $90\\%$ recall@1 and recall@10 in just around one millisecond with only 32GB memory cost. Code is available at: {\\footnotesize\\color{blue}{\\url{https://github.com/microsoft/SPTAG}}}.

Motivation & Objective

• Address the high memory cost of in-memory approximate nearest neighbor search (ANNS) for billion-scale vector datasets.
• Develop a hybrid ANNS system that uses minimal memory and inexpensive SSDs to support large-scale vector search.
• Achieve low query latency and high recall by minimizing disk I/O and improving posting list quality.
• Enable efficient distributed deployment with workload balancing to avoid hot spots and ensure scalability.
• Demonstrate superior performance and scalability compared to existing disk-based ANNS solutions like DiskANN and HM-ANN.

Proposed method

• Use hierarchical balanced clustering to partition vectors and balance posting list lengths, improving load distribution and reducing disk access.
• Augment posting lists by including points in the closure of their clusters to enhance search quality without increasing memory cost.
• Store only centroid points in memory and full posting lists on disk, minimizing memory footprint while maintaining high recall.
• Apply a query-aware dynamic pruning technique during search to reduce the number of disk-accessed posting lists and limit I/O and CPU cost.
• Deploy a best-fit bin-packing algorithm to balance data size and historical query access patterns across machines, preventing hot spots in distributed settings.
• Use a distributed architecture where each machine holds a partition of the index, and queries are routed to only the most relevant machines based on query-aware pruning.

Experimental results

Research questions

• RQ1 Can a simple inverted index-based approach achieve state-of-the-art performance in billion-scale approximate nearest neighbor search with minimal memory usage?
• RQ2 How can disk I/O be minimized while maintaining high recall in a memory-disk hybrid ANNS system?
• RQ3 What techniques can effectively improve posting list quality without increasing memory cost?
• RQ4 Can query-aware dynamic pruning significantly reduce the number of machines involved in each search, improving latency and scalability?
• RQ5 How can workload distribution be balanced across machines in a distributed ANNS system to avoid hot spots and ensure linear scalability?

Key findings

• SPANN achieves 2× faster search than DiskANN at 90% recall on three billion-scale datasets, using the same memory budget.
• SPANN reaches 90% recall@1 and recall@10 in approximately 1 millisecond with only 32GB of memory, reducing memory cost by 90% compared to baseline.
• The system reduces the average number of machines dispatched per query from 32 (random partition) to 6.3 through multi-constraint clustering and query-aware pruning, saving 80.3% in I/O and CPU cost.
• SPANN demonstrates strong scalability in distributed settings, with query access and data size evenly distributed across machines, enabling linear scaling with added resources.
• The use of closure assignment in posting list expansion improves search quality, contributing to higher recall without increasing memory footprint.
• SPANN has been successfully deployed in Microsoft Bing to support hundreds of billions of vector searches, confirming real-world efficiency and robustness.