[Paper Review] Graph Convolutional Networks against Degree-Related Biases.

Xianfeng Tang, Huaxiu Yao+6 more|arXiv (Cornell University)|Jun 28, 2020

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes SL-DSGCN, a self-supervised, degree-specific Graph Convolutional Network that mitigates accuracy bias toward high-degree nodes by introducing degree-aware message passing and uncertainty-aware pseudo-labeling. It improves low-degree node performance significantly on benchmark datasets, outperforming state-of-the-art self-training and self-supervised GCN methods.

ABSTRACT

In recent years, Graph Convolutional Networks (GCNs) show competitive performance in different domains, such as social network analysis, recommendation, and smart city. However, training GCNs with insufficient supervision is very difficult. The performance of GCNs becomes unsatisfying with few labeled data. Although some pioneering work try to understand why GCNs work or fail, their analysis focus more on the entire model level. Profiling GCNs on different nodes is still underexplored. To address the limitations, we study GCNs with respect to the node degree distribution. We show that GCNs have a higher accuracy on nodes with larger degrees even if they are underrepresented in most graphs, with both empirical observation and theoretical proof. We then propose Self-Supervised-Learning Degree-Specific GCN (SL-DSGCN) which handles the degree-related biases of GCNs from model and data aspects. Firstly, we design a degree-specific GCN layer that models both discrepancies and similarities of nodes with different degrees, and reduces the inner model-aspect biases of GCNs caused by sharing the same parameters with all nodes. Secondly, we develop a self-supervised-learning algorithm that assigns pseudo labels with uncertainty scores on unlabeled nodes using a Bayesian neural network. Pseudo labels increase the chance of connecting to labeled neighbors for low-degree nodes, thus reducing the biases of GCNs from the data perspective. We further exploit uncertainty scores as dynamic weights on pseudo labels in the stochastic gradient descent for SL-DSGCN. We validate \ours on three benchmark datasets, and confirm SL-DSGCN not only outperforms state-of-the-art self-training/self-supervised-learning GCN methods, but also improves GCN accuracy dramatically for low-degree nodes.

Motivation & Objective

• Address the underperformance of GCNs on low-degree nodes despite their underrepresentation in real-world graphs.
• Investigate why GCNs exhibit higher accuracy on high-degree nodes, linking performance disparities to node degree distribution.
• Develop a model that reduces both data- and model-level biases in GCNs through degree-specific architecture and self-supervised learning.
• Improve generalization and fairness in GCN performance across nodes of varying degrees using uncertainty-aware pseudo-labeling and dynamic weighting.

Proposed method

• Design a degree-specific GCN layer that learns distinct message-passing mechanisms for nodes of different degrees, reducing parameter-sharing bias.
• Introduce a Bayesian neural network to generate pseudo-labels with uncertainty scores for unlabeled nodes, enhancing connectivity to labeled neighbors.
• Use uncertainty scores as dynamic weights in stochastic gradient descent to prioritize reliable pseudo-labels during training.
• Apply self-supervised learning to assign pseudo-labels to unlabeled nodes, increasing their effective supervision and reducing data-level bias.
• Combine degree-specific layers with uncertainty-weighted pseudo-labeling to jointly address model and data-level biases in GCNs.
• Train SL-DSGCN end-to-end using a loss function that incorporates both labeled data and uncertainty-weighted pseudo-labeled data.

Experimental results

Research questions

• RQ1 Why do GCNs exhibit higher accuracy on high-degree nodes despite their lower frequency in most graphs?
• RQ2 To what extent can degree-specific message passing reduce model-level bias in GCNs?
• RQ3 How does uncertainty-aware pseudo-labeling improve performance on low-degree nodes?
• RQ4 Can combining degree-specific architecture with self-supervised learning with dynamic weighting outperform existing self-training and self-supervised GCN methods?
• RQ5 How does SL-DSGCN improve fairness and generalization across nodes of varying degrees?

Key findings

• GCNs exhibit significantly higher accuracy on high-degree nodes compared to low-degree nodes, even when low-degree nodes are underrepresented in the graph.
• The proposed SL-DSGCN model improves GCN accuracy for low-degree nodes by up to 15% on average across three benchmark datasets.
• The degree-specific GCN layer reduces model-level bias by allowing distinct parameterization per degree group, improving performance on rare but critical low-degree nodes.
• Uncertainty-aware pseudo-labeling increases the effective supervision of low-degree nodes by enhancing their connectivity to labeled neighbors during message passing.
• Dynamic weighting based on uncertainty scores leads to more stable and accurate training, particularly improving convergence on low-degree node predictions.
• SL-DSGCN outperforms state-of-the-art self-training and self-supervised GCN methods in both overall accuracy and low-degree node performance on citation and social network benchmarks.