[Paper Review] GNNDelete: A General Strategy for Unlearning in Graph Neural Networks

Jiali Cheng, George Dasoulas+3 more|arXiv (Cornell University)|Feb 26, 2023

Advanced Graph Neural Networks4 citations

TL;DR

GNNDelete proposes a model-agnostic, layer-wise unlearning operator for graph neural networks that enables efficient deletion of nodes, edges, and node features while preserving model performance. It enforces Deleted Edge Consistency and Neighborhood Influence through a learnable deletion weight matrix, achieving up to 38.8% higher AUC on edge deletion tasks and reducing unlearning time by 12.3x compared to retraining from scratch.

ABSTRACT

Graph unlearning, which involves deleting graph elements such as nodes, node labels, and relationships from a trained graph neural network (GNN) model, is crucial for real-world applications where data elements may become irrelevant, inaccurate, or privacy-sensitive. However, existing methods for graph unlearning either deteriorate model weights shared across all nodes or fail to effectively delete edges due to their strong dependence on local graph neighborhoods. To address these limitations, we introduce GNNDelete, a novel model-agnostic layer-wise operator that optimizes two critical properties, namely, Deleted Edge Consistency and Neighborhood Influence, for graph unlearning. Deleted Edge Consistency ensures that the influence of deleted elements is removed from both model weights and neighboring representations, while Neighborhood Influence guarantees that the remaining model knowledge is preserved after deletion. GNNDelete updates representations to delete nodes and edges from the model while retaining the rest of the learned knowledge. We conduct experiments on seven real-world graphs, showing that GNNDelete outperforms existing approaches by up to 38.8% (AUC) on edge, node, and node feature deletion tasks, and 32.2% on distinguishing deleted edges from non-deleted ones. Additionally, GNNDelete is efficient, taking 12.3x less time and 9.3x less space than retraining GNN from scratch on WordNet18.

Motivation & Objective

• To address the lack of effective, efficient unlearning methods for graph neural networks that can delete nodes, edges, and node features without retraining.
• To ensure that deleted graph elements no longer influence model predictions, satisfying the principle of exact unlearning.
• To preserve the model's predictive performance on remaining data by minimizing unintended changes to neighborhood representations.
• To develop a general-purpose, model-agnostic unlearning strategy applicable to any GNN architecture.
• To reduce computational cost and memory usage compared to full retraining, especially on large graphs.

Proposed method

• Introduces a layer-wise unlearning operator that learns a deletion weight matrix $W_D$ to update node representations after deletion.
• Enforces Deleted Edge Consistency by ensuring the model's prediction for deleted edges becomes random (AUC ≈ 0.5), indicating no residual influence.
• Enforces Neighborhood Influence by minimizing changes to the representations of nodes in the local neighborhood of deleted elements.
• Optimizes a composite loss function combining both consistency and influence objectives to balance unlearning accuracy and model fidelity.
• Applies the deletion operator only to the layers affected by the deleted elements, enabling efficient inference without full retraining.
• Supports deletion of nodes, node labels, and edges, and is compatible with various GNN architectures including GCN and GAT.

Experimental results

Research questions

• RQ1 Can a general-purpose unlearning method be designed for GNNs that effectively removes the influence of deleted graph elements?
• RQ2 How can unlearning be achieved without degrading model performance on remaining data?
• RQ3 Can the method maintain high unlearning accuracy while significantly reducing time and memory costs compared to retraining?
• RQ4 Does the method preserve the causal structure of the graph by minimizing changes to neighboring node representations?
• RQ5 How does the method perform across diverse real-world graphs and different types of deletions (edges, nodes, features)?

Key findings

• GNNDelete achieves up to 38.8% higher AUROC on edge deletion tasks compared to the next best baseline, with 0.976 AUROC on WordNet18 under full deletion.
• On distinguishing deleted edges from non-deleted ones, GNNDelete improves performance by 32.2% over existing methods.
• The method reduces unlearning time by 12.3x and memory usage by 9.3x compared to full retraining on the WordNet18 dataset.
• GNNDelete maintains high performance on node-level and graph-level tasks, with AUROC scores consistently above 0.93 on most benchmarks.
• The method outperforms retraining in terms of unlearning accuracy on edge deletion tasks, even though retraining is the gold standard.
• The ablation study confirms that both Deleted Edge Consistency and Neighborhood Influence are essential for high performance, with ablation on either reducing AUROC by over 10%.