[Paper Review] Representation Learning over Dynamic Graphs

Rakshit Trivedi, Mehrdad Farajtabar+2 more|arXiv (Cornell University)|Mar 11, 2018

Advanced Graph Neural Networks30 references47 citations

TL;DR

DyRep is an inductive deep framework that learns time-evolving node embeddings in dynamic graphs by jointly modeling and updating association (topology) and communication (interactions) events via a time-scale aware temporal point process and mediation-based embedding updates.

ABSTRACT

How can we effectively encode evolving information over dynamic graphs into low-dimensional representations? In this paper, we propose DyRep, an inductive deep representation learning framework that learns a set of functions to efficiently produce low-dimensional node embeddings that evolves over time. The learned embeddings drive the dynamics of two key processes namely, communication and association between nodes in dynamic graphs. These processes exhibit complex nonlinear dynamics that evolve at different time scales and subsequently contribute to the update of node embeddings. We employ a time-scale dependent multivariate point process model to capture these dynamics. We devise an efficient unsupervised learning procedure and demonstrate that our approach significantly outperforms representative baselines on two real-world datasets for the problem of dynamic link prediction and event time prediction.

Motivation & Objective

• Motivate the need for principled representation learning on dynamic graphs with two interacting processes: association (topology growth) and communication (interactions).
• Propose a unified, inductive framework to learn time-evolving node embeddings that mediate the two processes.
• Develop a time-scale dependent point process model and deep embedding updates to capture nonlinear dynamics at multiple temporal scales.
• Ensure scalability to large numbers of events through an efficient learning procedure.

Proposed method

• Define two temporal processes on graphs: Association (topology growth) and Communication (interactions).
• Introduce DyRep with three core functions: Temporal Function (multi time-scale conditional intensity via a neural network), Embedding Update Function (deep recurrent update), and Attentive Aggregate Function (intensity-augmented attention).
• Use a compatibility score g_k(t) = ω_k^T [z^u(t); z^v(t)] to parameterize the conditional intensity λ_k^{u,v}(t) with a time-scale dependent softplus: f_k(x) = ψ_k log(1+exp(x/ψ_k)).
• Update node embeddings with a three-term update: localized embedding propagation from neighbor u, self-propagation from the node itself, and exogenous drive based on time elapsed (Eq. 4).
• Employ an intensity-based attention mechanism to aggregate neighbor information via a shared stochastic matrix S and 1-hop neighborhoods (Equations 5 and related).
• Train by maximizing the log-likelihood of observed events with an efficient surrogate for the survival term to achieve scalability (Eq. 7 and Algorithm 2).

Experimental results

Research questions

• RQ1 Can a single inductive framework learn time-evolving node embeddings that capture both topology changes and interaction dynamics in dynamic graphs?
• RQ2 Does a time-scale aware temporal point process combined with mediation-based embedding updates outperform baselines for dynamic link prediction and event time prediction?
• RQ3 Can an intensity-based attention mechanism effectively propagate mesoscopic information through a graph to update embeddings?
• RQ4 Is the proposed learning procedure scalable to large numbers of events while maintaining predictive accuracy?
• RQ5 How do dynamic link prediction and event time prediction perform across real-world datasets with different network properties?

Key findings

• DyRep significantly outperforms representative baselines on dynamic link prediction and event time prediction in two real-world datasets.
• The model handles multi-scale dynamics by learning time-scale parameters per event type and using a softplus-based intensity function.
• Embedding updates combine localized propagation, self-propagation, and exogenous drive to reflect event-driven changes.
• An intensity-based attention mechanism enables mesoscopic information propagation through local neighborhoods.
• The learning procedure scales linearly with the number of events, enabling web-scale dynamic networks.
• DyRep demonstrates strong performance on both social evolution and Github datasets for both communication and association events.