[Paper Review] Graph Information Bottleneck

Tailin Wu, Hongyu Ren+2 more|arXiv (Cornell University)|Oct 24, 2020

Advanced Graph Neural Networks52 references96 citations

TL;DR

Graph Information Bottleneck (GIB) introduces an information-theoretic principle for graph representation learning that compresses both graph structure and node features while maximizing predictive information, yielding robust GNNs against adversarial attacks via two instantiations: GIB-Cat and GIB-Bern.

ABSTRACT

Representation learning of graph-structured data is challenging because both graph structure and node features carry important information. Graph Neural Networks (GNNs) provide an expressive way to fuse information from network structure and node features. However, GNNs are prone to adversarial attacks. Here we introduce Graph Information Bottleneck (GIB), an information-theoretic principle that optimally balances expressiveness and robustness of the learned representation of graph-structured data. Inheriting from the general Information Bottleneck (IB), GIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target, and simultaneously constraining the mutual information between the representation and the input data. Different from the general IB, GIB regularizes the structural as well as the feature information. We design two sampling algorithms for structural regularization and instantiate the GIB principle with two new models: GIB-Cat and GIB-Bern, and demonstrate the benefits by evaluating the resilience to adversarial attacks. We show that our proposed models are more robust than state-of-the-art graph defense models. GIB-based models empirically achieve up to 31% improvement with adversarial perturbation of the graph structure as well as node features.

Motivation & Objective

• Motivate robust representation learning on graphs by balancing expressiveness and robustness.
• Extend the Information Bottleneck to graph-structured data by regularizing both structure and feature information.
• Provide tractable variational bounds and practical instantiations for GNNs under the GIB principle.
• Demonstrate robustness gains over baselines and defense models against adversarial attacks.

Proposed method

• Formulate Graph Information Bottleneck (GIB) to minimize I(D; ZX) while maximizing I(Y; ZX) under a local-dependence assumption on graphs.
• Derive variational upper and lower bounds for the GIB objective and instantiate them with two sampling schemes.
• Instantiate GIB on Graph Attention Networks (GAT) resulting in two models: GIB-Cat (categorical neighbor sampling) and GIB-Bern (Bernoulli neighbor sampling).
• Use two sampling algorithms to refine the graph structure via attention weights and perform message passing over the refined structure.
• Train with a bound-based objective that combines AIB and XIB terms and a cross-entropy-like term for Y given ZX.
• Ensure permutation invariance of the learned representations.

Experimental results

Research questions

• RQ1 Can GIB improve the robustness of graph representations against adversarial perturbations in graph structure and node features?
• RQ2 How do structural information (AIB) and feature information (XIB) contribute to robustness in GNNs?
• RQ3 Do GIB-Cat and GIB-Bern provide practical, scalable defenses compared to existing graph defense models?
• RQ4 What is the impact of using local-dependence-based GIB in place of i.i.d. assumptions for graph data?

Key findings

• GIB-Cat and GIB-Bern consistently outperform baselines like GCN and GAT in adversarial settings across Cora and PubMed datasets.
• Under adversarial perturbations, GIB models achieve up to 31.3% (GIB-Cat) and 34.0% (GIB-Bern) improvements on targeted nodes in PubMed with fewer perturbations.
• Ablation shows both AIB (structure) and XIB (features) contribute to robustness, with structural AIB playing a significant role against Nettack attacks.
• GIB models demonstrate robustness to feature perturbations and are invariant to node permutations.
• In Citeseer, GIB methods may underperform against specialized attacks like GCNJaccard due to dataset-specific degree distributions.