[Paper Review] Node2Seq: Towards Trainable Convolutions in Graph Neural Networks

Hao Yuan, Shuiwang Ji|arXiv (Cornell University)|Jan 6, 2021

Advanced Graph Neural Networks40 references5 citations

TL;DR

This paper proposes Node2Seq, a novel graph neural network layer that enables trainable, adaptive convolutional weights for neighboring nodes via attention-based ordering and 1D convolutional networks. By incorporating learnable attention scores and adaptive non-local information, Node2Seq achieves state-of-the-art performance on node classification benchmarks, outperforming GCN, GAT, and other baselines with consistent improvements across datasets.

ABSTRACT

Investigating graph feature learning becomes essentially important with the emergence of graph data in many real-world applications. Several graph neural network approaches are proposed for node feature learning and they generally follow a neighboring information aggregation scheme to learn node features. While great performance has been achieved, the weights learning for different neighboring nodes is still less explored. In this work, we propose a novel graph network layer, known as Node2Seq, to learn node embeddings with explicitly trainable weights for different neighboring nodes. For a target node, our method sorts its neighboring nodes via attention mechanism and then employs 1D convolutional neural networks (CNNs) to enable explicit weights for information aggregation. In addition, we propose to incorporate non-local information for feature learning in an adaptive manner based on the attention scores. Experimental results demonstrate the effectiveness of our proposed Node2Seq layer and show that the proposed adaptively non-local information learning can improve the performance of feature learning.

Motivation & Objective

• To address the limitation of fixed or non-explicit neighbor weighting in existing graph neural networks like GCN and GAT.
• To enable learnable, dynamic convolutional weights for neighboring nodes during feature aggregation.
• To mitigate information loss from ignoring distant or non-local nodes in graph representation learning.
• To develop a mechanism that adaptively incorporates non-local node information based on attention scores.
• To improve node feature learning performance through a novel, trainable convolutional layer in GNNs.

Proposed method

• Order neighboring nodes using an attention mechanism to create a learnable sequence for each target node.
• Apply 1D convolutional neural networks (CNNs) on the ordered neighbor features to learn explicit, trainable weights for each neighbor.
• Integrate a global readout function to aggregate all neighbor features into a unified representation, preserving information from all neighbors regardless of degree.
• Adaptively incorporate non-local node information by learning a dynamic weighting factor based on attention scores.
• Combine the Node2Seq layer with residual connections and stack multiple layers to build a deep GNN architecture.
• Use a learnable weight matrix and non-linear activation functions to update node representations through message passing.

Experimental results

Research questions

• RQ1 Can learnable, convolutional weights for neighboring nodes improve node representation learning in GNNs?
• RQ2 How does attention-based neighbor ordering affect the performance of graph neural networks?
• RQ3 To what extent does incorporating non-local information enhance node classification accuracy?
• RQ4 How do different kernel sizes in the 1D CNN component affect model performance?
• RQ5 Does the proposed adaptive non-local mechanism consistently improve performance across diverse graph datasets?

Key findings

• Node2Seq outperforms GCN, GAT, and other baselines on all six benchmark datasets, achieving a 84.0% ± 0.6 accuracy on Cora-F, compared to 81.6% for GCN.
• The model with adaptive non-local information (Node2Seq_g) significantly outperforms the local-only variant (Node2Seq_l), demonstrating the value of non-local context.
• On the Squirrel dataset, Node2Seq achieves 58.8% ± 1.4 accuracy, surpassing the 56.5% of the GCN-baseline model with the same architecture.
• The optimal kernel size varies by dataset: k=3 yields the best performance on Squirrel, while performance on Cora improves with larger kernels up to k=10.
• Ablation studies confirm that the performance gain is primarily due to the Node2Seq layer, not architectural design, as replacing it with a GCN layer leads to performance degradation on three datasets.
• The proposed method maintains competitive performance across all kernel sizes, indicating robustness and generalization capability.