[Paper Review] DynGEM: Deep Embedding Method for Dynamic Graphs

Palash Goyal, Nitin Kamra+2 more|arXiv (Cornell University)|May 29, 2018

Advanced Graph Neural Networks22 references288 citations

TL;DR

DynGEM is a dynamic graph embedding method that incrementally learns stable, scalable node embeddings over growing graphs using a deep autoencoder and layer-expansion heuristic, outperforming static baselines on reconstruction, link prediction, and stability.

ABSTRACT

Embedding large graphs in low dimensional spaces has recently attracted significant interest due to its wide applications such as graph visualization, link prediction and node classification. Existing methods focus on computing the embedding for static graphs. However, many graphs in practical applications are dynamic and evolve constantly over time. Naively applying existing embedding algorithms to each snapshot of dynamic graphs independently usually leads to unsatisfactory performance in terms of stability, flexibility and efficiency. In this work, we present an efficient algorithm DynGEM based on recent advances in deep autoencoders for graph embeddings, to address this problem. The major advantages of DynGEM include: (1) the embedding is stable over time, (2) it can handle growing dynamic graphs, and (3) it has better running time than using static embedding methods on each snapshot of a dynamic graph. We test DynGEM on a variety of tasks including graph visualization, graph reconstruction, link prediction and anomaly detection (on both synthetic and real datasets). Experimental results demonstrate the superior stability and scalability of our approach.

Motivation & Objective

• Motivate stable embedding of dynamic, growing graphs for downstream tasks like visualization, link prediction, and anomaly detection.
• Develop a scalable deep autoencoder framework that incrementally trains across time steps by reusing previous embeddings.
• Enable growth handling by dynamically expanding network architecture while preserving prior weights.
• Introduce stability metrics to quantify embedding consistency over time and compare against baselines.

Proposed method

• Use a deep autoencoder to map node neighborhoods to a low-dimensional embedding (first- and second-order proximities).
• Initialize embeddings at time t from embeddings at time t-1 to promote temporal stability.
• Handle growing graphs with a PropSize heuristic to determine new layer sizes and Net2WiderNet/Net2DeeperNet to expand the network.
• Train with a loss L_net = L_glob + α L_loc + ν1 L1 + ν2 L2, balancing reconstruction and proximity objectives.
• Employ ReLU activations for scalability and faster training, with Nesterov momentum and regularization to prevent overfitting.
• Evaluate stability using a defined stability constant and assess scalability against static baselines.

Experimental results

Research questions

• RQ1 Can DynGEM provide stable embeddings over time for dynamic, growing graphs?
• RQ2 Does reusing previous time-step embeddings improve training efficiency and scalability?
• RQ3 How does DynGEM perform on graph reconstruction and link prediction compared to static embeddings and GF-based baselines?
• RQ4 Is the method useful for visualization and anomaly detection in dynamic networks?

Key findings

• DynGEM achieves high graph reconstruction MAP on SYN, ENRON, and AS datasets, e.g., DynGEM MAPs: SYN 0.987, ENRON 0.424; DynGEM often surpasses GF-based baselines and is competitive with or better than SDNE on several datasets.
• DynGEM significantly improves link prediction MAP across datasets (e.g., SYN 0.194, HEP-TH 0.26, AS 0.21, ENRON 0.084) compared to GF baselines and SDNE variants.
• Stability: DynGEM attains substantially lower stability constants than SDNE-based baselines (e.g., SYN 0.008 vs SDNE 0.18, HEP-TH 1.469 vs SDNE 14.715, AS 0.125 vs 6.25, ENRON 1.279 vs 19.722).
• Layer expansion via PropSize with Net2WiderNet/Net2DeeperNet yields better performance than fixed architectures (HEP-TH: reconstruction MAP improves from 0.46 to 0.491 and link prediction from 0.19 to 0.26).
• DynGEM provides 2-3x speedups over SDNE-aligned baselines across datasets, and scales well with longer time series (T=40 shown).
• DynGEM enables visualization of dynamic embeddings and anomaly detection (e.g., Enron weeks with spikes correspond to known events; embedding deltas Δt detect anomalies).