[Paper Review] Graph Representation Learning via Graphical Mutual Information Maximization

Zhen Peng, Wenbing Huang+5 more|arXiv (Cornell University)|Feb 4, 2020

Advanced Graph Neural NetworksComputer Science41 references45 citations

TL;DR

The paper introduces Graphical Mutual Information (GMI) to unsupervisedly learn graph embeddings by maximizing the mutual information between input graphs (features and topology) and node-level outputs, using a FMI-based decomposition and MINE-based estimation, showing strong performance on node classification and link prediction.

ABSTRACT

The richness in the content of various information networks such as social networks and communication networks provides the unprecedented potential for learning high-quality expressive representations without external supervision. This paper investigates how to preserve and extract the abundant information from graph-structured data into embedding space in an unsupervised manner. To this end, we propose a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graphs and high-level hidden representations. GMI generalizes the idea of conventional mutual information computations from vector space to the graph domain where measuring mutual information from two aspects of node features and topological structure is indispensable. GMI exhibits several benefits: First, it is invariant to the isomorphic transformation of input graphs---an inevitable constraint in many existing graph representation learning algorithms; Besides, it can be efficiently estimated and maximized by current mutual information estimation methods such as MINE; Finally, our theoretical analysis confirms its correctness and rationality. With the aid of GMI, we develop an unsupervised learning model trained by maximizing GMI between the input and output of a graph neural encoder. Considerable experiments on transductive as well as inductive node classification and link prediction demonstrate that our method outperforms state-of-the-art unsupervised counterparts, and even sometimes exceeds the performance of supervised ones.

Motivation & Objective

• Motivate unsupervised graph representation learning without labels to exploit rich content and linkage information in graphs.
• Define a graph-domain mutual information objective that is invariant to graph isomorphisms and scalable for node-level learning.
• Decompose graphical mutual information into tractable local mutual informations between node features and neighbors, with topology-aware weighting.
• Propose a practical encoder–decoder-like framework that optimizes GMI to produce expressive node embeddings for downstream tasks.

Proposed method

• Define Feature Mutual Information (FMI) between node embeddings and neighbor features and show a decomposition that yields a weighted sum of local MIs.
• Introduce Graphical Mutual Information (GMI) by extending FMI with topology-aware weights w_ij and a second term I(w_ij; a_ij) to connect weights to edge features.
• Use Mutual Information Neural Estimation (MINE) with a JSD-based objective to estimate I(h_i; x_j) at node level, enabling tractable optimization.
• Adopt a bilinear discriminator D(h_i, x_j) for MI estimation and allow trainable attention-like weights w_ij = σ(h_i^T h_j) to reflect topological proximity.
• Provide two implementation variants: GMI-adaptive (w_ij = σ(h_i^T h_j)) and GMI-mean (constant weights 1/i_n).
• Utilize a standard GCN encoder and train end-to-end to maximize the GMI objective for transductive and inductive tasks.

Experimental results

Research questions

• RQ1 How can mutual information be defined and maximized between input graphs and node-level embeddings in an unsupervised manner?
• RQ2 Can a graph-domain MI objective that incorporates both node features and topology yield superior representations compared to existing unsupervised methods?
• RQ3 Does a FMI-based decomposition coupled with topology-aware weighting enable scalable, permutation-invariant graph representation learning?
• RQ4 Do GMI-based embeddings achieve competitive or superior performance on node classification and link prediction across various datasets (transductive and inductive) compared to baselines?

Key findings

• GMI provides a topologically invariant, tractable objective by decomposing mutual information into a weighted sum of local MIs plus a topology term.
• Maximizing GMI with MINE (via a JSD estimator) yields expressive node embeddings usable for both node classification and link prediction.
• Experimental results show that the proposed method outperforms state-of-the-art unsupervised methods and can surpass some supervised approaches on several benchmarks.
• The approach supports both transductive and inductive settings and demonstrates robustness across diverse graph types (citation, social, and protein networks).
• Two variants (GMI-adaptive and GMI-mean) offer a trade-off between adaptive weighting and simplicity, with competitive performance in experiments.