[Paper Review] GraphZip: Dictionary-based Compression for Mining Graph Streams

Charles Packer, Lawrence B. Holder|arXiv (Cornell University)|Mar 24, 2017

Data Mining Algorithms and Applications28 references3 citations

TL;DR

GraphZip proposes a dictionary-based compression method inspired by Lempel-Ziv algorithms to mine maximally-compressing subgraph patterns from graph streams. By applying the minimum description length principle, it efficiently identifies complex, insightful patterns in real-time, outperforming state-of-the-art methods in speed and scalability on large real-world and synthetic graphs.

ABSTRACT

A massive amount of data generated today on platforms such as social networks, telecommunication networks, and the internet in general can be represented as graph streams. Activity in a network's underlying graph generates a sequence of edges in the form of a stream; for example, a social network may generate a graph stream based on the interactions (edges) between different users (nodes) over time. While many graph mining algorithms have already been developed for analyzing relatively small graphs, graphs that begin to approach the size of real-world networks stress the limitations of such methods due to their dynamic nature and the substantial number of nodes and connections involved. In this paper we present GraphZip, a scalable method for mining interesting patterns in graph streams. GraphZip is inspired by the Lempel-Ziv (LZ) class of compression algorithms, and uses a novel dictionary-based compression approach in conjunction with the minimum description length principle to discover maximally-compressing patterns in a graph stream. We experimentally show that GraphZip is able to retrieve complex and insightful patterns from large real-world graphs and artificially-generated graphs with ground truth patterns. Additionally, our results demonstrate that GraphZip is both highly efficient and highly effective compared to existing state-of-the-art methods for mining graph streams.

Motivation & Objective

• To address the scalability limitations of conventional graph mining algorithms on massive, dynamic real-world graphs such as social networks and telecommunication systems.
• To enable efficient, real-time analysis of graph streams where edges arrive continuously and cannot be stored in main memory.
• To discover interesting, highly compressing subgraph patterns by leveraging compression as a proxy for pattern significance.
• To develop a scalable, streaming-compatible graph mining framework that operates within strict time and memory constraints.
• To demonstrate that compression-driven pattern discovery via dictionary-based methods can effectively retrieve complex and meaningful substructures from large-scale graph streams.

Proposed method

• GraphZip uses a dictionary-based compression approach inspired by the Lempel-Ziv (LZ) family of algorithms, where recurring subgraph patterns are replaced by compact symbols.
• It applies the minimum description length (MDL) principle to identify subgraphs that yield maximal compression of the graph stream.
• The algorithm processes graph streams in batches, identifying all subgraph isomorphisms (embeddings) of existing dictionary patterns in each batch using the VF2 algorithm for subgraph isomorphism.
• For each embedding, it extends the pattern by one edge in each direction and adds the new extended pattern to the dictionary if it improves compression.
• Remaining edges not covered by existing patterns are added as single-edge patterns to the dictionary.
• The method dynamically builds a pattern dictionary that captures the most compressing substructures, which serve as proxies for interesting patterns in the graph stream.

Experimental results

Research questions

• RQ1 Can a compression-based approach effectively identify meaningful and complex subgraph patterns in large, dynamic graph streams?
• RQ2 How does the performance of GraphZip compare to state-of-the-art streaming and non-streaming graph mining methods in terms of speed and accuracy?
• RQ3 To what extent can GraphZip process graph streams in real time, given the constraints of limited memory and single-pass processing?
• RQ4 Does the use of the minimum description length principle lead to the discovery of patterns that are both highly compressing and semantically insightful?
• RQ5 How do parameter choices such as batch size and dictionary size affect the time complexity and scalability of the algorithm?

Key findings

• GraphZip successfully retrieves complex and insightful patterns from large real-world graphs, including the Twitter graph stream, at speeds far exceeding the actual stream rate of 0.93 tweets per second.
• In experiments on artificially-generated graphs with ground truth patterns, GraphZip achieved comparable accuracy to a state-of-the-art method but with significantly greater speed.
• GraphZip processed the full Twitter graph stream in under 80 hours, while the stream rate was only 0.93 tweets per second, indicating real-time feasibility.
• The algorithm outperformed both streaming and non-streaming state-of-the-art methods in terms of both efficiency and effectiveness on diverse real-world datasets.
• Theoretical complexity analysis shows that the algorithm's runtime is theoretically O(θ² × α!), but in practice, with fixed parameters, it scales linearly with graph size due to constant-time complexity per batch.
• The dominant computational cost stems from subgraph isomorphism checks, highlighting the need for approximation algorithms in future work to further improve efficiency.