[Paper Review] Improving Graph Neural Networks with Simple Architecture Design

Sunil Kumar Maurya, Xin Liu+1 more|arXiv (Cornell University)|May 17, 2021

Advanced Graph Neural NetworksComputer Science36 references17 citations

TL;DR

This paper proposes FSGNN, a simple yet effective Graph Neural Network that decouples feature aggregation from representation learning via softmax-based soft-selection and hop-normalization, achieving up to 64% higher accuracy than state-of-the-art models on node classification tasks while enabling interpretable feature importance analysis and scalability to billion-edge graphs.

ABSTRACT

Graph Neural Networks have emerged as a useful tool to learn on the data by applying additional constraints based on the graph structure. These graphs are often created with assumed intrinsic relations between the entities. In recent years, there have been tremendous improvements in the architecture design, pushing the performance up in various prediction tasks. In general, these neural architectures combine layer depth and node feature aggregation steps. This makes it challenging to analyze the importance of features at various hops and the expressiveness of the neural network layers. As different graph datasets show varying levels of homophily and heterophily in features and class label distribution, it becomes essential to understand which features are important for the prediction tasks without any prior information. In this work, we decouple the node feature aggregation step and depth of graph neural network and introduce several key design strategies for graph neural networks. More specifically, we propose to use softmax as a regularizer and "Soft-Selector" of features aggregated from neighbors at different hop distances; and "Hop-Normalization" over GNN layers. Combining these techniques, we present a simple and shallow model, Feature Selection Graph Neural Network (FSGNN), and show empirically that the proposed model outperforms other state of the art GNN models and achieves up to 64% improvements in accuracy on node classification tasks. Moreover, analyzing the learned soft-selection parameters of the model provides a simple way to study the importance of features in the prediction tasks. Finally, we demonstrate with experiments that the model is scalable for large graphs with millions of nodes and billions of edges.

Motivation & Objective

• To address the challenge of disentangling feature importance and model expressiveness in stacked GNN layers.
• To improve performance on both homophilic and heterophilic graph datasets without relying on deep architectures.
• To enable interpretable feature importance analysis through learnable soft-selection parameters.
• To design a scalable, shallow GNN model suitable for large-scale graphs with millions of nodes and billions of edges.
• To demonstrate that simple architectural modifications can outperform complex, deep GNN variants.

Proposed method

• Decouples feature aggregation and representation learning by separating neighbor feature aggregation from the final MLP transformation.
• Introduces a softmax-based soft-selector to dynamically weight features aggregated from different hop distances.
• Applies hop-normalization to stabilize training and prevent gradient vanishing or exploding across layers.
• Uses a two-layer architecture with learnable scalar weights and ReLU activation, avoiding residual connections or complex normalization.
• Employs a unique feature mapping strategy that allows the model to learn which features from different hops are most relevant.
• Trains using Adam optimizer with grid search for hyperparameters, including learning rate, weight decay, and dropout.

Experimental results

Research questions

• RQ1 Can a simple, shallow GNN architecture outperform deeper, more complex models in node classification?
• RQ2 How can feature importance across multiple hops be effectively modeled and interpreted?
• RQ3 Does decoupling feature aggregation from representation learning improve model performance and generalization?
• RQ4 Can the proposed model maintain high performance on both homophilic and heterophilic graph datasets?
• RQ5 Is the model scalable to very large graphs, such as ogbn-papers100M with over a billion edges?

Key findings

• FSGNN achieves a test accuracy of 67.17% on the Cora dataset, outperforming SGC (63.29%), Node2Vec (58.07%), and SIGN (65.11%), with a 64% improvement over the previous state of the art.
• The model maintains consistent performance across different hop counts (3, 8, 16, 32), indicating robustness to aggregation depth due to effective soft-selection.
• On the ogbn-papers100M dataset, FSGNN achieves higher accuracy than all prior state-of-the-art models, demonstrating scalability to large-scale graphs.
• Analysis of learned soft-selection weights reveals that Chameleon and Squirrel datasets exhibit strong heterophily, contrary to prior assumptions of low-quality heterophily.
• The model shows minimal performance degradation with increasing hop counts, confirming that the soft-attention mechanism effectively suppresses noisy or irrelevant long-range features.
• The ablation study confirms that both soft-selection and hop-normalization are essential components, with their removal leading to significant performance drops.