[Paper Review] MM-GNN: Mix-Moment Graph Neural Network towards Modeling Neighborhood Feature Distribution

Wendong Bi, Lun Du+4 more|arXiv (Cornell University)|Aug 15, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes MM-GNN, a novel Graph Neural Network that models neighborhood feature distribution using multi-order moments instead of single statistics (e.g., mean, max). By combining Multi-order Moment Embedding and an Element-wise Attention-based Moment Adaptor, MM-GNN adaptively weights higher-order moments, achieving state-of-the-art performance on 15 real-world graphs, including social and citation networks.

ABSTRACT

Graph Neural Networks (GNNs) have shown expressive performance on graph representation learning by aggregating information from neighbors. Recently, some studies have discussed the importance of modeling neighborhood distribution on the graph. However, most existing GNNs aggregate neighbors' features through single statistic (e.g., mean, max, sum), which loses the information related to neighbor's feature distribution and therefore degrades the model performance. In this paper, inspired by the method of moment in statistical theory, we propose to model neighbor's feature distribution with multi-order moments. We design a novel GNN model, namely Mix-Moment Graph Neural Network (MM-GNN), which includes a Multi-order Moment Embedding (MME) module and an Element-wise Attention-based Moment Adaptor module. MM-GNN first calculates the multi-order moments of the neighbors for each node as signatures, and then use an Element-wise Attention-based Moment Adaptor to assign larger weights to important moments for each node and update node representations. We conduct extensive experiments on 15 real-world graphs (including social networks, citation networks and web-page networks etc.) to evaluate our model, and the results demonstrate the superiority of MM-GNN over existing state-of-the-art models.

Motivation & Objective

• Address the limitation of existing GNNs that rely on single-order statistics (e.g., mean, max) for neighborhood aggregation, which fails to capture full feature distribution.
• Investigate the importance of higher-order moment statistics (beyond mean and variance) in distinguishing node representations on graphs.
• Design a GNN architecture that explicitly models multi-order moments of neighbor features to preserve richer distributional information.
• Introduce an adaptive attention mechanism to dynamically assign importance to different moment orders per node, improving representation learning.
• Demonstrate the effectiveness and robustness of the proposed method across diverse graph types and varying training data ratios.

Proposed method

• Propose Multi-order Moment Embedding (MME) to compute finite-order moments (up to order K) of neighbor features for each node, capturing distributional characteristics.
• Design an Element-wise Attention-based Moment Adaptor (AMA) that learns node-specific attention weights over the moment signatures to emphasize informative orders.
• Integrate MME and AMA into a message-passing framework, allowing the model to update node representations using weighted, multi-order moment signals.
• Use a learnable attention mechanism that operates element-wise across moment orders, enabling per-node, adaptive selection of relevant distributional features.
• Train the model end-to-end using standard GNN optimization, with the moment aggregation replacing traditional single-statistic aggregators.
• Limit moment computation to a finite, configurable maximum order K, balancing expressiveness and computational cost.

Experimental results

Research questions

• RQ1 Can modeling higher-order moments of neighborhood features improve node representation learning in GNNs compared to single-statistic aggregation?
• RQ2 How do different moment orders (e.g., mean, variance, skewness) contribute to distinguishing nodes with similar first-order statistics?
• RQ3 Does an adaptive attention mechanism over moment orders lead to better performance and interpretability than fixed or uniform weighting?
• RQ4 How does MM-GNN perform across diverse graph types, including social networks, citation networks, and web-page networks?
• RQ5 How robust is MM-GNN to variations in training data ratio and hyperparameters such as the maximum moment order K?

Key findings

• MM-GNN achieves state-of-the-art performance on 15 real-world graphs, including social networks (e.g., Facebook, Chameleon), citation networks (e.g., Cora, PubMed), and web-page networks (e.g., Flickr).
• On the Facebook social network, MM-GNN maintains superior performance even with only 10% training data, demonstrating strong generalization and robustness.
• Performance improves with increasing maximum moment order K, stabilizing when K ≥ 3, indicating that higher-order moments (e.g., skewness, kurtosis) carry meaningful information.
• Attention visualization shows that social networks assign higher weights to higher-order moments, while citation networks favor lower-order moments, aligning with data complexity and distribution patterns.
• The attention mechanism is interpretable: higher attention on a moment order correlates with better performance of the single-moment baseline for that order, validating its adaptive selection capability.
• MM-GNN outperforms strong baselines like GCN, GIN, GraphSAGE, GCNII, and DAGNN across all evaluated datasets and tasks, confirming the value of multi-order moment modeling.