[Paper Review] Subgraph Federated Learning with Missing Neighbor Generation

Ke Zhang, Carl Yang+3 more|arXiv (Cornell University)|Jun 25, 2021

Advanced Graph Neural NetworksComputer Science42 references75 citations

TL;DR

This paper introduces FedSage and FedSage+ for subgraph federated learning, enabling global graph knowledge transfer across distributed subgraphs, and addressing missing cross-subgraph neighbors with a generated missing-neighbor module.

ABSTRACT

Graphs have been widely used in data mining and machine learning due to their unique representation of real-world objects and their interactions. As graphs are getting bigger and bigger nowadays, it is common to see their subgraphs separately collected and stored in multiple local systems. Therefore, it is natural to consider the subgraph federated learning setting, where each local system holds a small subgraph that may be biased from the distribution of the whole graph. Hence, the subgraph federated learning aims to collaboratively train a powerful and generalizable graph mining model without directly sharing their graph data. In this work, towards the novel yet realistic setting of subgraph federated learning, we propose two major techniques: (1) FedSage, which trains a GraphSage model based on FedAvg to integrate node features, link structures, and task labels on multiple local subgraphs; (2) FedSage+, which trains a missing neighbor generator along FedSage to deal with missing links across local subgraphs. Empirical results on four real-world graph datasets with synthesized subgraph federated learning settings demonstrate the effectiveness and efficiency of our proposed techniques. At the same time, consistent theoretical implications are made towards their generalization ability on the global graphs.

Motivation & Objective

• Motivate learning a globally applicable graph classifier from multiple biased subgraphs without sharing raw data.
• Propose FedSage to aggregate GraphSage models via FedAvg across local subgraphs.
• Address cross-subgraph missing links with FedSage+ by generating missing neighbors.
• Demonstrate effectiveness and efficiency on real-world datasets under various data-owner settings.
• Provide theoretical insights on generalization in the GNTK framework for federated subgraph learning.

Proposed method

• Adopt FedAvg to train a shared GraphSage classifier across M local subgraphs (FedSage).
• Define an ego-graph based loss for node classification and propagate representations through K GraphSage layers.
• Introduce NeighGen, a two-module generator (encoder H^e and generator H^g) to generate missing cross-subgraph neighbors.
• Train NeighGen via graph-mending on impaired local subgraphs and jointly optimize with GraphSage (LocSage+).
• Federate NeighGen by exchanging gradients related to cross-subgraph neighbor reconstruction to enhance diversity of generated neighbors without sharing raw data.
• Provide a joint loss L = L^n + λ^c L^c for LocSage+ and a cross-subgraph limited FL scheme to update NeighGen locally with gradient signals from other owners.

Experimental results

Research questions

• RQ1 How can a globally applicable graph classifier be learned from distributed, biased subgraphs without sharing data?
• RQ2 Can FedAvg with GraphSage produce a robust global model under missing cross-subgraph links?
• RQ3 Does generating missing neighbors (NeighGen) improve performance in subgraph federated learning?
• RQ4 How can NeighGen be trained in a federated setting to capture cross-subgraph information without exposing private data?
• RQ5 What are the generalization implications of subgraph FL under the GNTK framework?

Key findings

• FedSage and FedSage+ consistently outperform locally trained classifiers across four real-world datasets.
• FedSage+ further improves performance over FedSage by mitigating the cross-subgraph missing-neighbor problem.
• FedSage+ shows robustness to cross-subgraph information loss, especially on sparser graphs like Citeseer.
• Local training without collaboration (LocSage/LocSage+) underperforms compared to federated approaches, illustrating the value of FL in this setting.
• Joint training of NeighGen and GraphSage yields faster convergence and competitive accuracy with manageable training time overhead.
• Hyper-parameter studies indicate that an α around 1 and moderate missing-neighbor rates h improve performance, while extremes degrade learning.