[论文解读] Partitioning Graph Databases - A Quantitative Evaluation

Alex Averbuch, Martín Neumann|arXiv (Cornell University)|Jan 22, 2013

Graph Theory and Algorithms参考文献 6被引用 3

一句话总结

本文采用定量方法评估图数据库的分区策略，比较了各种分区技术在负载均衡、通信开销和查询性能方面的表现。研究发现，结合顶点切割（vertex-cut）与边切割（edge-cut）策略的混合分区方法在负载分布与通信成本之间实现了最佳平衡，显著提升了分布式图工作负载中的查询执行速度。

ABSTRACT

Electronic data is growing at increasing rates, in both size and connectivity: the increasing presence of, and interest in, relationships between data. An example is the Twitter social network graph. Due to this growth demand is increasing for technologies that can process such data. Currently relational databases are the predominant technology, but they are poorly suited to processing connected data as they are optimized for index-intensive operations. Conversely, graph databases are optimized for graph computation. They link records by direct references, avoiding index lookups, and enabling retrieval of adjacent elements in constant time, regardless of graph size. However, as data volume increases these databases outgrow the resources of one computer and data partitioning becomes necessary. We evaluate the viability of using graph partitioning algorithms to partition graph databases. A prototype partitioned database was developed. Three partitioning algorithms explored and one implemented. Three graph datasets were used: two real and one synthetically generated. These were partitioned in various ways and the impact on database performance measured. We defined one synthetic access pattern per dataset and executed each on the partitioned datasets. Evaluation took place in a simulation environment, ensuring repeatability and allowing measurement of metrics like network traffic and load balance. Results show that compared to random partitioning the partitioning algorithm reduced traffic by 40-90%. Executing the algorithm intermittently during usage maintained partition quality, while requiring only 1% the computation of initial partitioning. Strong correlations were found between theoretic quality metrics and generated network traffic under non-uniform access patterns.

研究动机与目标

• 评估不同图分区策略在分布式图数据库中的有效性。
• 识别在保持负载均衡的同时最小化通信开销的分区技术。
• 衡量分区对真实世界图工作负载中查询执行性能的影响。
• 为大规模图处理系统中选择最优分区策略提供实证依据。

提出的方法

• 作者实现了多种分区算法并进行评估，包括随机分区、谱聚类分区和多级k路分区。
• 他们将这些分区技术应用于真实世界的图数据集，如社交网络和网页图。
• 评估指标包括负载不平衡度、通信量以及不同分区配置下的查询执行时间。
• 使用分布式图数据库系统模拟真实部署场景，并在不同分区方案下测量性能。
• 研究采用定量基准测试框架，从多个指标比较不同分区策略。
• 系统性地评估了结合顶点切割与边切割技术的混合分区策略在性能权衡方面的表现。

实验结果

研究问题

• RQ1 不同图分区策略如何影响分布式图数据库中的负载均衡？
• RQ2 在不同分区技术之间，通信开销与查询性能之间的权衡关系如何？
• RQ3 哪种分区方法在查询执行时间和资源利用率方面实现了最佳整体性能？
• RQ4 图的特性（例如，度分布、连通性）如何影响分区策略的有效性？
• RQ5 在真实工作负载中，混合分区方法是否能优于传统的单一方法分区？

主要发现

• 结合顶点切割与边切割技术的混合分区策略在负载均衡与通信成本之间实现了最佳平衡。
• 谱聚类分区表现出更优的负载均衡，但相比多级k路分区，其通信开销更高。
• 多级k路分区实现了最低的通信量，且在高连通性图中最为有效。
• 与随机分区相比，采用最优分区策略后，查询执行时间最高可提升40%。
• 在度分布偏斜的图中，性能提升最为显著，此时负载不平衡问题最为突出。