[Paper Review] Certifiably Robust Graph Contrastive Learning

Minhua Lin, Teng Xiao+3 more|arXiv (Cornell University)|Oct 5, 2023

Advanced Graph Neural NetworksComputer Science3 citations

TL;DR

This paper proposes RES (Randomized Edgedrop Smoothing), the first certifiably robust framework for Graph Contrastive Learning (GCL), by introducing a unified robustness criterion based on semantic similarity in latent space and injecting randomized edge-drop noise to ensure provable robustness. It achieves 43.96% certified accuracy on OGB-arxiv under structural attacks with up to 10 added edges, outperforming baselines and demonstrating transferable robustness to downstream tasks.

ABSTRACT

Graph Contrastive Learning (GCL) has emerged as a popular unsupervised graph representation learning method. However, it has been shown that GCL is vulnerable to adversarial attacks on both the graph structure and node attributes. Although empirical approaches have been proposed to enhance the robustness of GCL, the certifiable robustness of GCL is still remain unexplored. In this paper, we develop the first certifiably robust framework in GCL. Specifically, we first propose a unified criteria to evaluate and certify the robustness of GCL. We then introduce a novel technique, RES (Randomized Edgedrop Smoothing), to ensure certifiable robustness for any GCL model, and this certified robustness can be provably preserved in downstream tasks. Furthermore, an effective training method is proposed for robust GCL. Extensive experiments on real-world datasets demonstrate the effectiveness of our proposed method in providing effective certifiable robustness and enhancing the robustness of any GCL model. The source code of RES is available at https://github.com/ventr1c/RES-GCL.

Motivation & Objective

• To address the lack of certifiable robustness in unsupervised Graph Contrastive Learning (GCL), which remains vulnerable to structural and attribute-based adversarial attacks.
• To develop a unified, label-free robustness evaluation criterion for GCL based on semantic similarity in the latent representation space.
• To enable provable robustness certification for GCL models without relying on labels or introducing excessive noisy edges during data augmentation.
• To ensure that certified robustness in GCL translates into robust performance in downstream tasks through theoretical and empirical validation.
• To design an effective training method that integrates randomized edgedrop noise to enhance robustness during GCL pre-training.

Proposed method

• Proposes a unified robustness criterion based on semantic similarity between node or graph representations in the latent space, enabling consistent evaluation without labels.
• Introduces Randomized Edgedrop Smoothing (RES), a novel technique that stochastically drops edges during inference to generate smoothed predictions and provide certified robustness.
• Employs Monte Carlo sampling with μ samples to estimate the probability bounds of predictions under perturbations, enabling certification via Theorem 2.
• Designs a training method that incorporates randomized edgedrop noise during pre-training to improve robustness, enhancing the model’s resilience to adversarial perturbations.
• Uses a confidence level (1−α) to control the robustness certificate threshold, with higher values yielding looser but more reliable bounds.
• Applies the framework to GRACE as a base GCL model, demonstrating transferability of certified robustness to downstream tasks.

Experimental results

Research questions

• RQ1 Can a unified, label-free robustness criterion be defined for Graph Contrastive Learning that consistently evaluates robustness across different downstream tasks?
• RQ2 Can randomized edgedrop smoothing provide provable robustness certification for GCL models without introducing excessive spurious edges?
• RQ3 Does certifiable robustness in GCL translate into robust performance in downstream classification tasks?
• RQ4 How does the proposed RES method compare to vanilla randomized smoothing in terms of robustness and downstream performance?
• RQ5 What is the impact of key hyperparameters (μ, β, 1−α) on the certified accuracy and robustness of the RES framework?

Key findings

• RES achieves 43.96% certified accuracy on the OGB-arxiv dataset under an attacker budget of 10 added edges, significantly outperforming baseline methods.
• RES consistently outperforms FLIP and FLIP/T—variants using binary random noise—on both clean and noisy graphs in Cora and Pubmed, validating its superiority in robustness and representation quality.
• Increasing the number of Monte Carlo samples (μ) improves certified accuracy, as tighter probability bounds lead to larger certified perturbation radii.
• Higher confidence levels (1−α) result in slightly lower certified accuracy due to looser probability bounds, reducing the number of nodes that satisfy the certification condition.
• The proposed training method with injected edgedrop noise enhances the robustness of GCL models, as evidenced by improved robust accuracy under structural attacks like Nettack.
• Theoretical analysis confirms that the robustness of the encoder trained with RES is provably preserved in downstream tasks, bridging the gap between unsupervised GCL and downstream robustness.