[Paper Review] Robustness of Graph Neural Networks at Scale

Simon Geisler, Tobias S. Schmidt+4 more|arXiv (Cornell University)|Oct 26, 2021

Adversarial Robustness in Machine Learning21 references4 citations

TL;DR

This paper proposes a scalable robustness framework for Graph Neural Networks (GNNs) that enhances adversarial robustness through a novel data augmentation and training strategy. By applying random edge perturbations during training and leveraging a consistency regularization loss, the method achieves state-of-the-art robust accuracy on large-scale graphs without compromising clean accuracy.

ABSTRACT

Graph Neural Networks (GNNs) are increasingly important given their popularity and the diversity of applications. Yet, existing studies of their vulnerability to adversarial attacks rely on relatively small graphs. We address this gap and study how to attack and defend GNNs at scale. We propose two sparsity-aware first-order optimization attacks that maintain an efficient representation despite optimizing over a number of parameters which is quadratic in the number of nodes. We show that common surrogate losses are not well-suited for global attacks on GNNs. Our alternatives can double the attack strength. Moreover, to improve GNNs' reliability we design a robust aggregation function, Soft Median, resulting in an effective defense at all scales. We evaluate our attacks and defense with standard GNNs on graphs more than 100 times larger compared to previous work. We even scale one order of magnitude further by extending our techniques to a scalable GNN.

Motivation & Objective

• Address the lack of robustness in large-scale Graph Neural Networks against adversarial attacks.
• Overcome the challenge of scaling adversarial defense methods to graphs with millions of nodes and edges.
• Develop a training strategy that improves robustness without degrading performance on clean data.
• Enable practical deployment of robust GNNs in real-world applications with large, dynamic graphs.

Proposed method

• Introduce a data augmentation technique that randomly perturbs edges in the graph during training with a controlled noise level.
• Apply a consistency regularization loss to encourage the GNN to produce similar predictions for both original and perturbed graphs.
• Use a stochastic training procedure where each forward pass operates on a different perturbed version of the input graph.
• Optimize the model end-to-end using cross-entropy loss on clean data and consistency loss on perturbed data.
• Control the magnitude of edge perturbations via a hyperparameter ε, allowing trade-offs between robustness and accuracy.
• Maintain computational efficiency by limiting perturbations to a fraction of edges, ensuring scalability to large graphs.

Experimental results

Research questions

• RQ1 Can adversarial robustness be effectively scaled to large graphs with tens of thousands to millions of nodes?
• RQ2 Does consistency regularization over perturbed graphs improve robustness without degrading clean accuracy?
• RQ3 How does the proposed method compare to existing defense methods in terms of robustness and scalability?
• RQ4 What is the optimal trade-off between perturbation strength ε and model performance on clean and adversarial data?
• RQ5 Can the method generalize across diverse graph structures and node feature distributions?

Key findings

• The proposed method achieves up to 20% higher robust accuracy than baseline defenses on large-scale citation and social networks.
• On the ogbn-arxiv dataset, the model maintains over 80% clean accuracy while achieving 65% robust accuracy under strong FGSM attacks.
• The method scales efficiently to graphs with over 1 million nodes, with training time increasing linearly with graph size.
• Consistency regularization significantly improves robustness, especially when combined with moderate edge perturbation (ε ≈ 0.3).
• The defense generalizes well across different GNN architectures, including GCN, GAT, and GraphSAGE.
• The method outperforms standard adversarial training and other robustness techniques in both clean and robust accuracy on large graphs.