[Paper Review] DPAR: Decoupled Graph Neural Networks with Node-Level Differential Privacy

Qiuchen Zhang, H. Lee+4 more|arXiv (Cornell University)|Oct 10, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes DPAR, a novel framework for training Graph Neural Networks (GNNs) with rigorous node-level differential privacy by decoupling message passing from feature aggregation using differentially private personalized PageRank (DP-APPR). It combines DP-APPR with differentially private SGD (DP-SGD) to protect both node features and graph topology, achieving strong privacy guarantees while maintaining high model utility on real-world datasets.

ABSTRACT

Graph Neural Networks (GNNs) have achieved great success in learning with graph-structured data. Privacy concerns have also been raised for the trained models which could expose the sensitive information of graphs including both node features and the structure information. In this paper, we aim to achieve node-level differential privacy (DP) for training GNNs so that a node and its edges are protected. Node DP is inherently difficult for GNNs because all direct and multi-hop neighbors participate in the calculation of gradients for each node via layer-wise message passing and there is no bound on how many direct and multi-hop neighbors a node can have, so existing DP methods will result in high privacy cost or poor utility due to high node sensitivity. We propose a Decoupled GNN with Differentially Private Approximate Personalized PageRank (DPAR) for training GNNs with an enhanced privacy-utility tradeoff. The key idea is to decouple the feature projection and message passing via a DP PageRank algorithm which learns the structure information and uses the top-$K$ neighbors determined by the PageRank for feature aggregation. By capturing the most important neighbors for each node and avoiding the layer-wise message passing, it bounds the node sensitivity and achieves improved privacy-utility tradeoff compared to layer-wise perturbation based methods. We theoretically analyze the node DP guarantee for the two processes combined together and empirically demonstrate better utilities of DPAR with the same level of node DP compared with state-of-the-art methods.

Motivation & Objective

• To address the critical privacy risk in GNNs where trained models may leak sensitive information from node features and edges.
• To formally establish node-level differential privacy for GNNs, protecting both node features and their connected edges simultaneously.
• To overcome the challenge of high sensitivity in GNNs due to node correlations, which undermines standard DP methods.
• To design a two-stage training framework that decouples message passing from feature aggregation to enable effective DP application.
• To achieve a strong privacy-utility trade-off in GNN training under rigorous node-level DP guarantees.

Proposed method

• Decouple the GNN training process into two stages: message passing via personalized PageRank (APPR) and feature aggregation using DP-SGD.
• Propose differentially private APPR algorithms using the Gaussian mechanism and exponential mechanism to protect graph topology information.
• Apply DP-SGD to the feature aggregation stage, with gradient clipping and noise injection to protect node features.
• Use a clipping bound in DP-APPR to control sensitivity and reduce redundant noise in the APPR vectors.
• Integrate DP-APPR and DP-SGD in a unified framework to ensure end-to-end node-level differential privacy.
• Analyze privacy degradation from sampling processes dependent on DP-APPR results and mitigate it in the DPGNN algorithm.

Experimental results

Research questions

• RQ1 Can we achieve rigorous node-level differential privacy for GNNs that protects both node features and their connected edges?
• RQ2 How can we decouple message passing from feature aggregation to reduce sensitivity and enable effective DP application in GNNs?
• RQ3 What are the privacy-utility trade-offs when applying DP-APPR and DP-SGD to GNN training on real-world graphs?
• RQ4 How does the choice of clipping bounds in DP-APPR and DP-SGD affect model utility and privacy guarantees?
• RQ5 Can we design a framework that ensures formal node-level DP without compromising model performance on downstream tasks?

Key findings

• The proposed DPAR framework achieves strong node-level differential privacy by protecting both node features and their connected edges in GNNs.
• DP-APPR effectively decorrelates nodes and limits sensitivity, enabling DP-SGD to be applied with meaningful utility.
• Experiments on real-world datasets show that DPAR maintains high model accuracy under strict privacy budgets, demonstrating a favorable privacy-utility trade-off.
• The use of clipping bounds in DP-APPR (set to 0.01 and 0.001) reduces redundant noise and improves model performance.
• DP-SGD with a gradient norm clipping bound of 1 achieves stable optimization and effective privacy protection.
• The framework is the first to formally achieve node-level DP for GNNs that covers both features and edges, setting a new standard in private GNN training.