[Paper Review] Federated Graph Classification over Non-IID Graphs

Han Xie, Jing Ma+2 more|arXiv (Cornell University)|Jun 25, 2021

Privacy-Preserving Technologies in DataComputer Science48 references63 citations

TL;DR

The paper proposes GCFL, a gradient-based clustered federated learning framework for graph classification on non-IID graph data, with GCFL+ using gradient-sequence DTW clustering to better handle heterogeneity; experiments show consistent improvements over baselines across multiple datasets and domains.

ABSTRACT

Federated learning has emerged as an important paradigm for training machine learning models in different domains. For graph-level tasks such as graph classification, graphs can also be regarded as a special type of data samples, which can be collected and stored in separate local systems. Similar to other domains, multiple local systems, each holding a small set of graphs, may benefit from collaboratively training a powerful graph mining model, such as the popular graph neural networks (GNNs). To provide more motivation towards such endeavors, we analyze real-world graphs from different domains to confirm that they indeed share certain graph properties that are statistically significant compared with random graphs. However, we also find that different sets of graphs, even from the same domain or same dataset, are non-IID regarding both graph structures and node features. To handle this, we propose a graph clustered federated learning (GCFL) framework that dynamically finds clusters of local systems based on the gradients of GNNs, and theoretically justify that such clusters can reduce the structure and feature heterogeneity among graphs owned by the local systems. Moreover, we observe the gradients of GNNs to be rather fluctuating in GCFL which impedes high-quality clustering, and design a gradient sequence-based clustering mechanism based on dynamic time warping (GCFL+). Extensive experimental results and in-depth analysis demonstrate the effectiveness of our proposed frameworks.

Motivation & Objective

• Show that real-world graphs share properties that enable cross-dataset federated learning for graph classification.
• Quantify structure and feature heterogeneity across graph datasets and domains.
• Develop GCFL to dynamically cluster clients by gradient similarity and train cluster-specific GNNs.
• Improve clustering by using gradient sequences via GCFL+ with dynamic time warping.
• Demonstrate empirical gains over FedAvg and FedProx across single-dataset and multi-dataset settings.

Proposed method

• Use Graph Isomorphism Network (GIN)-based graph classification as the core model.
• Cluster clients dynamically by analyzing transmitted gradients to form homogeneous groups.
• Train cluster-specific GNN models via FedAvg within each cluster.
• GCFL introduces two stopping/partition criteria based on gradient norms to trigger clustering.
• GCFL+ maintains a gradient-norm time-series matrix and uses dynamic time warping to refine clustering over rounds.
• Provide theoretical justification that GNN gradients reflect graph structure and feature differences.

Experimental results

Research questions

• RQ1 Can gradient-based clustering reduce structure and feature heterogeneity in non-IID graph federated learning?
• RQ2 Do cluster-specific GNNs trained within GCFL outperform vanilla FedAvg and FedProx on graph classification tasks?
• RQ3 Does incorporating gradient-sequence information via DTW (GCFL+) yield more stable and better clusters than using the last-gradient alone?
• RQ4 Is cross-dataset/cross-domain graph Federated Learning beneficial for graph classification across heterogeneous sources?

Key findings

• GCFL and GCFL+ improve graph classification accuracy over self-training and standard FL baselines (FedAvg, FedProx) across multiple datasets.
• In single-dataset settings, GCFL/GCFL+ achieved average gains of up to about 14.75 percentage points over self-train on some datasets.
• Across multiple datasets and domains, GCFL/GCFL+ consistently improve a majority of clients, with GCFL+ often outperforming GCFL.
• GCFL+ uses gradient-sequence DTW clustering to better capture long-range training dynamics, improving clustering quality and performance.
• Theoretical results show GNN gradients bound weight changes with respect to structure/feature differences, supporting gradient-based clustering validity.
• GCFL+ maintains robustness by preventing poorly benefiting clients from dragging down the cluster through sequence-based clustering.