[论文解读] Navigable Proximity Graph-Driven Native Hybrid Queries with Structured and Unstructured Constraints

Mengzhao Wang, Lingwei Lv+4 more|arXiv (Cornell University)|Mar 25, 2022

Advanced Image and Video Retrieval Techniques被引用 4

一句话总结

本文提出了一种基于可导航邻近图（NPGs）的原生混合查询（NHQ）框架，该框架联合优化了向量相似性搜索与结构化属性过滤，以支持混合查询。通过集成专用复合索引与联合剪枝机制，NHQ框架在八个公开数据集和一个内部数据集上实现了比最先进方法快达10倍的性能，同时保持了高召回率。

ABSTRACT

As research interest surges, vector similarity search is applied in multiple fields, including data mining, computer vision, and information retrieval. {Given a set of objects (e.g., a set of images) and a query object, we can easily transform each object into a feature vector and apply the vector similarity search to retrieve the most similar objects. However, the original vector similarity search cannot well support extit{hybrid queries}, where users not only input unstructured query constraint (i.e., the feature vector of query object) but also structured query constraint (i.e., the desired attributes of interest). Hybrid query processing aims at identifying these objects with similar feature vectors to query object and satisfying the given attribute constraints. Recent efforts have attempted to answer a hybrid query by performing attribute filtering and vector similarity search separately and then merging the results later, which limits efficiency and accuracy because they are not purpose-built for hybrid queries.} In this paper, we propose a native hybrid query (NHQ) framework based on proximity graph (PG), which provides the specialized extit{composite index and joint pruning} modules for hybrid queries. We easily deploy existing various PGs on this framework to process hybrid queries efficiently. Moreover, we present two novel navigable PGs (NPGs) with optimized edge selection and routing strategies, which obtain better overall performance than existing PGs. After that, we deploy the proposed NPGs in NHQ to form two hybrid query methods, which significantly outperform the state-of-the-art competitors on all experimental datasets (10$ imes$ faster under the same extit{Recall}), including eight public and one in-house real-world datasets. Our code and datasets have been released at \url{https://github.com/AshenOn3/NHQ}.

研究动机与目标

• 解决现有混合查询处理方法在将向量相似性搜索与属性过滤视为独立步骤时存在的效率低下与不准确问题。
• 设计一个统一框架，原生支持单次查询执行中的非结构化（特征向量）与结构化（属性）约束。
• 开发优化的可导航邻近图（NPGs），通过改进的边选择与路由策略，提升在混合查询工作负载下的整体性能。
• 在真实世界与公开数据集中，显著优于现有最先进系统。

提出的方法

• 提出一种原生混合查询（NHQ）框架，利用邻近图（PG）结构将向量相似性搜索与属性过滤整合到单一联合处理流水线中。
• 引入一种复合索引结构，将特征向量与属性元数据在PG中结合，以实现高效的混合查询执行。
• 设计联合剪枝模块，在遍历过程中同时减少向量相似性与属性约束的搜索空间。
• 开发两种新型可导航PG（NPG），其边选择与路由策略针对混合查询进行了优化，从而提升整体检索性能。
• 在NHQ框架中部署所提出的NPG，形成两种混合查询方法：NHQ-NPG_kgraph与NHQ-NPG_sprawl。
• 在路由策略中引入动态权重调节机制，以平衡向量相似性与属性约束满足度。

实验结果

研究问题

• RQ1 能否设计一个统一框架，原生支持结合非结构化与结构化约束的混合查询，避免顺序处理带来的低效？
• RQ2 如何通过复合索引与联合剪枝增强邻近图，以提升混合查询性能？
• RQ3 在混合查询场景中，可导航邻近图的最优边选择与路由策略是什么？
• RQ4 所提出的NHQ框架能否在性能上显著超越现有最先进混合查询系统？

主要发现

• NHQ框架在所有评估数据集上实现了比最先进竞争对手快达10倍的性能提升，同时保持相同的召回率。
• 所提出的NHQ-NPG_kgraph与NHQ-NPG_sprawl方法在全部八个公开数据集与一个内部真实世界数据集上均优于现有系统。
• 两种新型NPG中优化的边选择与路由策略相比现有PG，带来了更优的整体性能。
• 联合剪枝与复合索引的使用减少了冗余搜索空间，提升了查询效率，且未牺牲准确性。
• 该框架与现有PG兼容，支持在混合查询处理中轻松部署多种PG。
• 在学术专家查找系统中的案例研究显示，当涉及属性约束时，NHQ-NPG_kgraph比HNSW与Vearch检索出更准确、更相关的专家。