[Paper Review] DropMessage: Unifying Random Dropping for Graph Neural Networks

Taoran Fang, Zhiqing Xiao+4 more|arXiv (Cornell University)|Apr 21, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes DropMessage, a novel random dropping method that operates directly on propagated messages in graph neural networks (GNNs), unifying existing methods under a single framework. By reducing sample variance and preserving information diversity, DropMessage stabilizes training, improves robustness, and achieves state-of-the-art performance across multiple datasets and models, with significant gains in accuracy and generalization under overfitting, over-smoothing, and noisy conditions.

ABSTRACT

Graph Neural Networks (GNNs) are powerful tools for graph representation learning. Despite their rapid development, GNNs also face some challenges, such as over-fitting, over-smoothing, and non-robustness. Previous works indicate that these problems can be alleviated by random dropping methods, which integrate augmented data into models by randomly masking parts of the input. However, some open problems of random dropping on GNNs remain to be solved. First, it is challenging to find a universal method that are suitable for all cases considering the divergence of different datasets and models. Second, augmented data introduced to GNNs causes the incomplete coverage of parameters and unstable training process. Third, there is no theoretical analysis on the effectiveness of random dropping methods on GNNs. In this paper, we propose a novel random dropping method called DropMessage, which performs dropping operations directly on the propagated messages during the message-passing process. More importantly, we find that DropMessage provides a unified framework for most existing random dropping methods, based on which we give theoretical analysis of their effectiveness. Furthermore, we elaborate the superiority of DropMessage: it stabilizes the training process by reducing sample variance; it keeps information diversity from the perspective of information theory, enabling it become a theoretical upper bound of other methods. To evaluate our proposed method, we conduct experiments that aims for multiple tasks on five public datasets and two industrial datasets with various backbone models. The experimental results show that DropMessage has the advantages of both effectiveness and generalization, and can significantly alleviate the problems mentioned above.

Motivation & Objective

• To address overfitting, over-smoothing, and non-robustness in GNNs, which are common challenges in graph representation learning.
• To unify existing random dropping methods—previously applied to features or adjacency matrices—into a single, coherent framework based on message-space operations.
• To provide theoretical justification for the effectiveness of random dropping in GNNs by linking it to regularization in the loss function.
• To improve training stability and model generalization by minimizing sample variance and preserving information diversity in messages.
• To demonstrate the superiority of DropMessage across diverse GNN architectures and real-world datasets, including industrial applications.

Proposed method

• DropMessage performs random dropping directly on the message matrix during the message-passing phase, allowing the same source node to send different messages to different neighbors.
• The method generalizes existing approaches by modeling them as special cases under its unified framework, where message masking follows specific rules.
• Theoretical analysis shows that message-space dropping introduces implicit regularization, improving model robustness and generalization.
• DropMessage maintains information diversity by ensuring that the expected mutual information between input and output messages remains high, making it a theoretical upper bound for other methods.
• The framework supports both node-wise and average-based dropping rates, with the former preserving information diversity in expectation.
• Experiments are conducted on five public and two industrial datasets using GCN, GAT, and APPNP as backbone models to evaluate performance across node classification, link prediction, and robustness tasks.

Experimental results

Research questions

• RQ1 Can a unified framework be established for existing random dropping methods in GNNs by operating on the message matrix rather than features or adjacency matrices?
• RQ2 How does message-space random dropping affect the stability and convergence of GNN training, particularly in terms of sample variance?
• RQ3 What is the theoretical relationship between random dropping in GNNs and regularization in the loss function?
• RQ4 To what extent does DropMessage preserve information diversity compared to other random dropping methods, and how does this impact model performance?
• RQ5 Does DropMessage outperform existing methods in mitigating overfitting, over-smoothing, and adversarial noise across diverse datasets and architectures?

Key findings

• DropMessage achieves an average improvement of 37% in robustness under 30% input perturbation compared to models without random dropping.
• On the Cora dataset, DropMessage improves MADGap by 3.3% and test accuracy by 4.9% on average for models with depth ≥3, outperforming other methods in alleviating over-smoothing.
• DropMessage exhibits the smallest sample variance in training loss, leading to faster convergence and more stable training compared to other random dropping techniques.
• In information diversity analysis, the node-wise dropping setting (where dropping rate is set to the theoretical upper bound) outperforms the average setting by 0.8–1.5% across all backbone models, confirming the importance of preserving diversity.
• DropMessage generalizes across multiple GNN architectures and datasets, showing consistent performance gains on both public benchmarks and two industrial datasets.
• Theoretical analysis confirms that message-space dropping introduces regularization, explaining its effectiveness in improving model robustness and generalization.