[Paper Review] Algorithm Engineering for High-Dimensional Similarity Search Problems (Invited Talk)

Masajiro Iwasaki|arXiv (Cornell University)|Oct 17, 2018

Advanced Image and Video Retrieval Techniques22 references39 citations

TL;DR

This paper proposes a graph-based indexing framework for high-dimensional similarity search that optimizes both indegree and outdegree of nodes to balance search accuracy and query efficiency. By introducing static, constrained, and dynamic degree adjustment methods, along with path optimization and visited-node management, the approach achieves faster query times and higher accuracy than prior methods on image and text datasets, with automatic parameter tuning for optimal performance.

ABSTRACT

Searching for high-dimensional vector data with high accuracy is an inevitable search technology for various types of data. Graph-based indexes are known to reduce the query time for high-dimensional data. To further improve the query time by using graphs, we focused on the indegrees and outdegrees of graphs. While a sufficient number of incoming edges (indegrees) are indispensable for increasing search accuracy, an excessive number of outgoing edges (outdegrees) should be suppressed so as to not increase the query time. Therefore, we propose three degree-adjustment methods: static degree adjustment of not only outdegrees but also indegrees, dynamic degree adjustment with which outdegrees are determined by the search accuracy users require, and path adjustment to remove edges that have alternative search paths to reduce outdegrees. We also show how to obtain optimal degree-adjustment parameters and that our methods outperformed previous methods for image and textual data.

Motivation & Objective

• To address the trade-off between search accuracy and query time in high-dimensional similarity search.
• To improve graph-based indexing by jointly optimizing indegrees and outdegrees, which are often treated independently in prior work.
• To enable dynamic adjustment of outdegree based on user-specified accuracy requirements.
• To reduce query time by eliminating redundant edges with alternative paths.
• To accelerate query processing through optimized visited-node management.

Proposed method

• Proposes static degree adjustment to control both indegree and outdegree using edges and reversed edges from a k-nearest neighbor graph (KNNG).
• Introduces constrained static degree adjustment to more precisely regulate indegree and outdegree with user-defined bounds.
• Develops dynamic degree adjustment that sets optimal outdegree at query time based on required accuracy, using a parameter ǫ to control exploration radius.
• Introduces path adjustment to remove shortcut edges that have alternative paths, reducing outdegrees and query cost.
• Improves visited-node management using a custom data structure to reduce overhead during graph traversal.
• Automatically tunes degree parameters (eo, ei) via empirical optimization to maximize performance on target datasets.

Experimental results

Research questions

• RQ1 Can jointly optimizing indegree and outdegree in a KNNG improve both search accuracy and query time?
• RQ2 How does dynamic degree adjustment based on required accuracy affect performance compared to static settings?
• RQ3 To what extent does path adjustment—removing redundant edges—reduce query time without sacrificing accuracy?
• RQ4 How effective is the proposed visited-node management in reducing query time overhead?
• RQ5 Can optimal degree parameters be automatically determined for different datasets to achieve consistent performance gains?

Key findings

• The proposed dynamic degree adjustment method outperformed static methods and prior approaches in terms of query time and accuracy, especially at high precision levels.
• Path adjustment significantly reduced outdegrees and query time by eliminating edges with alternative paths, without degrading search accuracy.
• The improved visited-node management reduced query time more effectively than standard containers (unordered map, array), particularly at high precision where initialization overhead was minimal.
• Optimal degree parameters (eo, ei) were found to be stable across different dataset sizes, suggesting that optimization on a subset can yield near-optimal results for larger datasets.
• The method achieved O(log n) computational complexity in search, indicating efficient scaling with dataset size.
• On SIFT 10M, the DA method outperformed ANNG, AKNNG, SAC, and PANNG in both number of distance computations and query time, demonstrating superior efficiency and accuracy.