[Paper Review] Variational Graph Recurrent Neural Networks

Ehsan Hajiramezanali, Arman Hasanzadeh+4 more|arXiv (Cornell University)|Aug 26, 2019

Advanced Graph Neural NetworksComputer Science26 references84 citations

TL;DR

The paper introduces VGRNN and SI-VGRNN, hierarchical variational models that embed dynamic graph nodes as latent random vectors to jointly model evolving topology and node attributes, with semi-implicit variational inference for flexible posteriors.

ABSTRACT

Representation learning over graph structured data has been mostly studied in static graph settings while efforts for modeling dynamic graphs are still scant. In this paper, we develop a novel hierarchical variational model that introduces additional latent random variables to jointly model the hidden states of a graph recurrent neural network (GRNN) to capture both topology and node attribute changes in dynamic graphs. We argue that the use of high-level latent random variables in this variational GRNN (VGRNN) can better capture potential variability observed in dynamic graphs as well as the uncertainty of node latent representation. With semi-implicit variational inference developed for this new VGRNN architecture (SI-VGRNN), we show that flexible non-Gaussian latent representations can further help dynamic graph analytic tasks. Our experiments with multiple real-world dynamic graph datasets demonstrate that SI-VGRNN and VGRNN consistently outperform the existing baseline and state-of-the-art methods by a significant margin in dynamic link prediction.

Motivation & Objective

• Motivate node embedding for dynamic graphs where topology and attributes evolve over time.
• Propose a dynamic graph autoencoder (GRNN) augmented with latent random variables to capture uncertainty.
• Increase expressive power by using high-level latent variables and flexible posteriors via semi-implicit variational inference.
• Enable future link prediction by learning time-conditioned priors and an autoregressive graph structure.

Proposed method

• Combine graph convolutional networks (GCN) with recurrent networks to form a graph recurrent neural network (GRNN).
• Extend GRNN with variational encoder/decoder to model node latent distributions (VGRNN).
• Condition priors on previous hidden state h_{t-1} and allow flexible priors via neural mappings φ^{prior}.
• Use an inner-product decoder p(A^{(t)}|Z^{(t)}) to generate adjacency with Z^{(t)} latent variables.
• Infer posteriors q(Z^{(t)}|A^{(t)},X^{(t)},h_{t-1}) via GNN-based encoders; include semi-implicit augmentation in SI-VGRNN.
• Optimize a variational lower bound (ELBO) across time, capturing temporal dependencies in dynamic graphs.

Experimental results

Research questions

• RQ1 How can dynamic graphs be represented with node embeddings that capture uncertainty and temporal evolution?
• RQ2 Does integrating stochastic latent variables with graph recurrent architectures improve dynamic link prediction compared to deterministic baselines?
• RQ3 Can semi-implicit variational inference provide more expressive posterior distributions for dynamic graph embeddings?
• RQ4 How does the proposed model handle changing node/edge sets over time and enable inductive and transductive tasks?

Key findings

• VGRNN and SI-VGRNN consistently outperform baselines and state-of-the-art methods on dynamic link prediction tasks across multiple real-world datasets.
• Introducing stochastic latent variables yields better modeling of variability and uncertainty in dynamic graphs than deterministic hidden states.
• SI-VGRNN's semi-implicit posteriors provide more flexible latent representations and can yield improvements over VGRNN, especially when node attributes are available.
• The benefits of our approach are more pronounced in sparser, less clustered graphs, where temporal trends are harder to capture with deterministic models.
• The prior construction based on past states enables future link prediction, including dynamic (new) link prediction, beyond static VGAE capabilities.