[Paper Review] GMNN: Graph Markov Neural Networks

Meng Qu, Yoshua Bengio+1 more|arXiv (Cornell University)|May 15, 2019

Advanced Graph Neural NetworksComputer Science146 citations

TL;DR

GMNN combines conditional random fields with graph neural networks in a variational EM framework to jointly model label dependencies and learn object representations for semi-supervised relational data.

ABSTRACT

This paper studies semi-supervised object classification in relational data, which is a fundamental problem in relational data modeling. The problem has been extensively studied in the literature of both statistical relational learning (e.g. relational Markov networks) and graph neural networks (e.g. graph convolutional networks). Statistical relational learning methods can effectively model the dependency of object labels through conditional random fields for collective classification, whereas graph neural networks learn effective object representations for classification through end-to-end training. In this paper, we propose the Graph Markov Neural Network (GMNN) that combines the advantages of both worlds. A GMNN models the joint distribution of object labels with a conditional random field, which can be effectively trained with the variational EM algorithm. In the E-step, one graph neural network learns effective object representations for approximating the posterior distributions of object labels. In the M-step, another graph neural network is used to model the local label dependency. Experiments on object classification, link classification, and unsupervised node representation learning show that GMNN achieves state-of-the-art results.

Motivation & Objective

• Motivate semi-supervised object classification on relational graphs and bridge SRL with GNNs.
• Model the joint label distribution using a conditional random field within a variational EM framework.
• Leverage two graph neural networks: one for inferring posterior label distributions and one for modeling local label dependencies.
• Avoid hand-crafted potentials by learning conditional distributions with neural networks.
• Demonstrate state-of-the-art or competitive performance on object classification, link classification, and unsupervised node representation tasks.

Proposed method

• Define p(y_V | x_V) with a conditional random field and optimize via variational EM.
• E-step: use a Graph Neural Network (GNN) to learn representations and approximate posterior p(y_U | y_L, x_V) via q_theta(y_U | x_V).
• M-step: use another GNN to model local label dependencies via p_phi(y_n | y_NB(n), x_V).
• Use mean-field approximation for inference and a pseudolikelihood objective to train p_phi; q_theta is amortized by a separate GNN.
• Alternate E-step and M-step, with q_theta typically used for final labeling.
• Optionally extend to unsupervised node representation learning and link classification by applying GMNN to line graphs or neighbor prediction tasks.

Experimental results

Research questions

• RQ1 Can a joint model combining SRL-style label dependencies with GNN-based representations improve semi-supervised object classification on relational graphs?
• RQ2 Does a variational EM training procedure with two GNNs (inference and dependency modeling) outperform end-to-end GNNs and traditional SRL methods?
• RQ3 How does GMNN perform on object classification, unsupervised node representation learning, and link classification compared to baselines?

Key findings

• GMNN achieves competitive to state-of-the-art results on object classification benchmarks (Cora, Citeseer, Pubmed).
• GMNN variants show improvements over baseline GNNs when incorporating label dependency in p_phi (with and without attributes).
• In unsupervised node representation learning, GMNN with both q_theta and p_phi matches or exceeds several strong baselines.
• On link classification tasks, GMNN demonstrates competitive performance using the line-graph transformation.
• The method generalizes to unsupervised and link-prediction tasks, in addition to standard semi-supervised node classification.