[Paper Review] Robust Causal Graph Representation Learning against Confounding Effects

Hang Gao, Jiangmeng Li+5 more|arXiv (Cornell University)|Aug 18, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes Robust Causal Graph Representation Learning (RCGRL), a novel method that generates instrumental variables under unconditional moment restrictions to eliminate confounding effects in graph representation learning. By actively identifying and removing confounders, RCGRL improves prediction performance and generalization on both in-domain and out-of-domain graph benchmarks, outperforming state-of-the-art methods across multiple datasets.

ABSTRACT

The prevailing graph neural network models have achieved significant progress in graph representation learning. However, in this paper, we uncover an ever-overlooked phenomenon: the pre-trained graph representation learning model tested with full graphs underperforms the model tested with well-pruned graphs. This observation reveals that there exist confounders in graphs, which may interfere with the model learning semantic information, and current graph representation learning methods have not eliminated their influence. To tackle this issue, we propose Robust Causal Graph Representation Learning (RCGRL) to learn robust graph representations against confounding effects. RCGRL introduces an active approach to generate instrumental variables under unconditional moment restrictions, which empowers the graph representation learning model to eliminate confounders, thereby capturing discriminative information that is causally related to downstream predictions. We offer theorems and proofs to guarantee the theoretical effectiveness of the proposed approach. Empirically, we conduct extensive experiments on a synthetic dataset and multiple benchmark datasets. The results demonstrate that compared with state-of-the-art methods, RCGRL achieves better prediction performance and generalization ability.

Motivation & Objective

• To address the overlooked issue that pre-trained graph models underperform on pruned graphs, revealing the presence of confounding substructures in graphs.
• To formalize the existence of graph confounders that degrade model performance and generalization by introducing spurious correlations.
• To develop a causal representation learning framework that actively removes confounders to learn representations causally linked to downstream predictions.
• To theoretically and empirically validate that instrumental variable generation under moment restrictions can effectively eliminate confounding effects in graph data.

Proposed method

• RCGRL introduces an active instrumental variable (IV) generation mechanism under unconditional moment restrictions to identify and mitigate confounding effects in graph data.
• The method applies IVs at a specific layer (u-th layer) of the GNN to remove confounders at the feature level, preserving semantic information while eliminating spurious correlations.
• It employs a weighted loss function (w) and a contrastive learning objective (ℒₐ) to enhance representation quality and robustness.
• The framework is grounded in causal inference theory, with theoretical guarantees provided through theorems and proofs for the validity of IV-based confounder removal.
• The model uses Graph Granger Causality to identify and measure confounders, enabling targeted pruning of non-causal substructures.
• RCGRL is trained end-to-end with a GNN backbone, integrating IV injection and contrastive learning to improve generalization on OOD and ID datasets.

Experimental results

Research questions

• RQ1 Do confounding substructures in graphs degrade the performance and generalization of graph neural networks?
• RQ2 Can instrumental variables generated under unconditional moment restrictions effectively eliminate confounders in graph representation learning?
• RQ3 Does injecting IVs at a specific layer of the GNN architecture preserve semantic information while removing spurious correlations?
• RQ4 How does RCGRL compare to state-of-the-art methods in terms of robustness on out-of-domain graph benchmarks?
• RQ5 What is the optimal position (layer) for injecting instrumental variables to maximize performance?

Key findings

• RCGRL outperforms all baselines, including ERM, GAT, Top-k Pool, Group DRO, IRM, and DIR, across all in-domain and out-of-domain graph benchmarks.
• On the Spurious-Motif dataset, RCGRL achieves a 12.3% improvement in accuracy over the best baseline, demonstrating strong generalization under distribution shift.
• The ablation study shows that removing the weight factor (w) or contrastive loss (ℒₐ) leads to performance drops, confirming their effectiveness in enhancing robustness.
• When instrumental variables are introduced at the u=2 layer, RCGRL achieves the best performance across all datasets, indicating optimal timing for confounder removal.
• Visualizations and confounder percentage analysis confirm that RCGRL reduces confounder proportion by over 50% on Mol-BACE and Mol-BBBP, without significant loss of semantic information.
• The method achieves state-of-the-art performance on Graph-SST2 (ID), Graph-Twitter, Mol-BBBP, and Mol-BACE, with consistent gains on both in- and out-of-distribution settings.