[Paper Review] EAGr: Supporting Continuous Ego-centric Aggregate Queries over Large Dynamic Graphs

Jayanta Mondal, Amol Deshpande|arXiv (Cornell University)|Jan 1, 2014

Data Management and Algorithms53 references8 citations

TL;DR

EAGr proposes an in-memory, single-machine framework for efficiently processing continuous ego-centric aggregate queries over large, dynamic graphs by using a pre-compiled aggregation overlay graph that enables shared partial computation and optimized push/pull dataflow decisions. The system achieves update and query throughputs exceeding 500,000 operations per second on graphs with up to 320 million nodes and edges.

ABSTRACT

In this work, we present EAGr, a system for supporting large numbers of continuous neighborhood-based ("ego-centric") aggregate queries over large, highly dynamic, and rapidly evolving graphs. Examples of such queries include computation of personalized, tailored trends in social networks, anomaly/event detection in financial transaction networks, local search and alerts in spatio-temporal networks, to name a few. Key challenges in supporting such continuous queries include high update rates typically seen in these situations, large numbers of queries that need to be executed simultaneously, and stringent low latency requirements. We propose a flexible, general, and extensible in-memory framework for executing different types of ego-centric aggregate queries over large dynamic graphs with low latencies. Our framework is built around the notion of an aggregation overlay graph, a pre-compiled data structure that encodes the computations to be performed when an update/query is received. The overlay graph enables sharing of partial aggregates across multiple ego-centric queries (corresponding to the nodes in the graph), and also allows partial pre-computation of the aggregates to minimize the query latencies. We present several highly scalable techniques for constructing an overlay graph given an aggregation function, and also design incremental algorithms for handling structural changes to the underlying graph. We also present an optimal, polynomial-time algorithm for making the pre-computation decisions given an overlay graph, and evaluate an approach to incrementally adapt those decisions as the workload changes. Although our approach is naturally parallelizable, we focus on a single-machine deployment and show that our techniques can easily handle graphs of size up to 320 million nodes and edges, and achieve update/query throughputs of over 500K/s using a single, powerful machine.

Motivation & Objective

• To address the challenge of low-latency, high-throughput processing of large numbers of continuous ego-centric aggregate queries over rapidly evolving, large-scale graphs.
• To reduce redundant computation across queries by exploiting sharing of partial aggregates across different ego networks.
• To design a scalable, extensible framework that supports user-defined aggregation functions and dynamic graph updates.
• To optimize dataflow decisions (push vs. pull) for minimal latency and maximal throughput.
• To enable efficient incremental updates to the overlay graph as the underlying graph structure evolves.

Proposed method

• Constructs an aggregation overlay graph—a directed, pre-compiled data structure—that encodes computation paths and dataflow decisions (push/pull) for each query.
• Uses scalable heuristics to build the overlay graph for arbitrary aggregation functions and neighborhood definitions (e.g., 1-hop, 2-hop, filtered neighborhoods).
• Employs a max-flow-based algorithm to compute optimal push/pull decisions for a given overlay, solvable in polynomial time.
• Supports incremental updates to the overlay graph in response to structural changes in the underlying graph, minimizing recomputation.
• Enables user-defined aggregates via a flexible API, supporting functions like sum, count, min, max, and top-k.
• Uses a single-machine, in-memory architecture with efficient memory management to achieve high throughput.

Experimental results

Research questions

• RQ1 How can we efficiently support continuous ego-centric aggregate queries over large, highly dynamic graphs with low latency?
• RQ2 What is the optimal way to structure an overlay graph to minimize redundant computation across multiple ego-centric queries?
• RQ3 Can we model and solve the dataflow decision problem (push vs. pull) in polynomial time for a given overlay?
• RQ4 How can the overlay be incrementally updated as the underlying graph evolves?
• RQ5 To what extent can sharing of partial aggregates and pre-computation improve throughput in real-time graph query processing?

Key findings

• The system achieves update and query throughputs exceeding 500,000 operations per second on a single machine with 64GB of memory.
• EAGr successfully scales to graphs with up to 320 million nodes and edges, demonstrating feasibility for real-world large-scale workloads.
• The optimal dataflow decision problem for a given overlay is solvable in polynomial time using a max-flow-based algorithm.
• The overlay construction heuristics scale effectively to large graphs and outperform prior graph compression techniques in terms of compression and efficiency.
• The framework significantly reduces redundant computation by sharing partial aggregates across multiple ego-centric queries.
• The system supports a wide range of aggregation functions and user-defined aggregates through a flexible API.