[Paper Review] Variational Graph Auto-Encoders

Thomas Kipf, Max Welling|arXiv (Cornell University)|Nov 21, 2016

Advanced Graph Neural Networks8 references897 citations

TL;DR

This paper introduces Variational Graph Auto-Encoders (VGAE), a probabilistic framework for unsupervised learning on graph-structured data using a graph convolutional encoder and inner-product decoder for link prediction, with features improving performance.

ABSTRACT

We introduce the variational graph auto-encoder (VGAE), a framework for unsupervised learning on graph-structured data based on the variational auto-encoder (VAE). This model makes use of latent variables and is capable of learning interpretable latent representations for undirected graphs. We demonstrate this model using a graph convolutional network (GCN) encoder and a simple inner product decoder. Our model achieves competitive results on a link prediction task in citation networks. In contrast to most existing models for unsupervised learning on graph-structured data and link prediction, our model can naturally incorporate node features, which significantly improves predictive performance on a number of benchmark datasets.

Motivation & Objective

• Develop a probabilistic latent variable model for unsupervised learning on undirected graphs.
• Leverage a two-layer GCN to parameterize the variational posterior over latent node embeddings.
• Train via a variational lower bound to learn meaningful latent representations for graphs.
• Demonstrate improved link prediction performance, especially when node features are available.
• Compare with baseline graph embedding methods and discuss effects of feature use and priors.

Proposed method

• Define a VGAE with latent per-node z_i and a Gaussian posterior q(z_i|X,A) parameterized by a two-layer GCN.
• Use a generative model p(A|Z) where A_ij|z_i,z_j ~ Bernoulli( sigmoid(z_i^T z_j) ).
• Optimize the variational lower bound L = E_{q(Z|X,A)}[log p(A|Z)] - KL[q(Z|X,A)||p(Z)], with p(Z)=N(0,I).
• Train with the reparameterization trick and full-batch gradient descent.
• Provide a non-probabilistic GAE variant using Z Z^T as the reconstruction for A via sigmoid.
• Experiment with featureful and featureless settings (X used vs identity).
• Compare VGAE/GAE against spectral clustering and DeepWalk baselines on link prediction.

Experimental results

Research questions

• RQ1 Can a variational approach learn meaningful latent embeddings for nodes in a graph in an unsupervised manner?
• RQ2 Does incorporating node features X improve link prediction performance over featureless variants?
• RQ3 How does VGAE/GAE compare to established baselines (spectral clustering, DeepWalk) on citation networks?
• RQ4 What is the impact of using probabilistic priors (Gaussian Z) with an inner-product decoder on performance?

Key findings

• VGAE and GAE achieve competitive results on link prediction in citation networks.
• Incorporating node features significantly improves predictive performance across datasets.
• Featureless variants (GAE*, VGAE*) perform reasonably but generally underperform feature-equipped models.
• GAE and VGAE with features outperform baselines on Cora, Citeseer, and Pubmed datasets in most metrics.
• A Gaussian prior may be suboptimal with an inner-product decoder, suggesting room for better priors or models.
• The models are trained with full-batch gradient descent and the reparameterization trick; future work includes scalability improvements.