[Paper Review] Time Constrained Continuous Subgraph Search over Streaming Graphs

Youhuan Li, Lei Zou+2 more|arXiv (Cornell University)|Jan 28, 2018

Graph Theory and Algorithms13 references3 citations

TL;DR

This paper proposes a novel system for continuous subgraph isomorphism search over streaming graphs with timing order constraints, using decomposition-based expansion lists and an MS-tree to reduce space and improve throughput. It achieves up to 10× higher throughput and significantly lower space cost than state-of-the-art methods, with effective concurrency management and real-world detection of C&C server communication patterns in network traffic.

ABSTRACT

The growing popularity of dynamic applications such as social networks provides a promising way to detect valuable information in real time. Efficient analysis over high-speed data from dynamic applications is of great significance. Data from these dynamic applications can be easily modeled as streaming graph. In this paper, we study the subgraph (isomorphism) search over streaming graph data that obeys timing order constraints over the occurrence of edges in the stream. We propose a data structure and algorithm to efficiently answer subgraph search and introduce optimizations to greatly reduce the space cost, and propose concurrency management to improve system throughput. Extensive experiments on real network traffic data and synthetic social streaming data confirms the efficiency and effectiveness of our solution.

Motivation & Objective

• Address the challenge of efficiently performing subgraph isomorphism search on high-speed streaming graph data with strict timing order constraints between edges.
• Reduce the high memory and computational cost of maintaining partial matches during continuous subgraph matching in dynamic graphs.
• Improve system throughput through effective concurrency management in subgraph matching over streaming data.
• Enable real-time detection of complex attack patterns (e.g., C&C communication, fraud) in network and social streaming data.
• Provide an exact subgraph isomorphism solution with timing constraints, unlike prior approximate or simulation-based approaches.

Proposed method

• Propose a decomposition strategy that splits the query graph into smaller, time-constrained subqueries (TC-subqueries) to manage partial matches efficiently.
• Introduce an expansion list data structure to maintain and update partial matches incrementally as new edges arrive in the stream.
• Design the MS-tree (Memory-Optimized Search Tree) to compress and index partial match states, drastically reducing space overhead.
• Implement concurrency management to allow parallel processing of multiple subquery matches, improving system throughput.
• Apply a join order strategy based on decomposition to minimize the number of intermediate partial matches and reduce computation cost.
• Use timestamp-based pruning and timing constraint validation to ensure only valid matches respecting edge order are retained.

Experimental results

Research questions

• RQ1 How can subgraph isomorphism with timing order constraints be efficiently maintained over high-speed streaming graphs?
• RQ2 What decomposition strategy minimizes the number of partial matches while preserving correctness and enabling efficient computation?
• RQ3 How can memory usage be reduced in continuous subgraph matching over streaming graphs without sacrificing performance?
• RQ4 What concurrency control mechanism enables high throughput in real-time subgraph matching over dynamic graphs?
• RQ5 Can the proposed system detect real-world attack patterns (e.g., C&C communication) in network traffic with high accuracy and low latency?

Key findings

• The proposed method achieves up to 10× higher throughput than existing state-of-the-art approaches on real and synthetic streaming graph workloads.
• The MS-tree reduces space cost significantly, with space usage growing only moderately as decomposition size (k) increases, confirming its memory efficiency.
• Throughput decreases with increasing decomposition size (k), confirming that smaller k values yield better performance, as expected from the design.
• The system successfully detected a ZeuS botnet C&C communication pattern in real network traffic, with matches occurring at the correct time (Nov. 28, 2017), validating real-world applicability.
• The method outperforms alternatives in both throughput and space efficiency across all tested workloads (network flow, social stream, wiki-talk), especially at larger window sizes.
• The selectivity of query results decreases with larger query size but increases with larger window size, reflecting the expected trade-off between pattern complexity and detection likelihood.