[Paper Review] Graph Pointer Neural Networks

Tianmeng Yang, Yujing Wang+4 more|arXiv (Cornell University)|Oct 3, 2021

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes Graph Pointer Neural Networks (GPNN), a novel GNN framework that uses a pointer network to selectively identify and rank the most relevant nodes—both local and remote—for each central node, enabling non-local, noise-resistant feature aggregation. By jointly optimizing the pointer network with end-to-end training, GPNN significantly reduces over-smoothing and improves node classification accuracy on heterophilic graphs, achieving a 6.3% average improvement over state-of-the-art methods.

ABSTRACT

Graph Neural Networks (GNNs) have shown advantages in various graph-based applications. Most existing GNNs assume strong homophily of graph structure and apply permutation-invariant local aggregation of neighbors to learn a representation for each node. However, they fail to generalize to heterophilic graphs, where most neighboring nodes have different labels or features, and the relevant nodes are distant. Few recent studies attempt to address this problem by combining multiple hops of hidden representations of central nodes (i.e., multi-hop-based approaches) or sorting the neighboring nodes based on attention scores (i.e., ranking-based approaches). As a result, these approaches have some apparent limitations. On the one hand, multi-hop-based approaches do not explicitly distinguish relevant nodes from a large number of multi-hop neighborhoods, leading to a severe over-smoothing problem. On the other hand, ranking-based models do not joint-optimize node ranking with end tasks and result in sub-optimal solutions. In this work, we present Graph Pointer Neural Networks (GPNN) to tackle the challenges mentioned above. We leverage a pointer network to select the most relevant nodes from a large amount of multi-hop neighborhoods, which constructs an ordered sequence according to the relationship with the central node. 1D convolution is then applied to extract high-level features from the node sequence. The pointer-network-based ranker in GPNN is joint-optimized with other parts in an end-to-end manner. Extensive experiments are conducted on six public node classification datasets with heterophilic graphs. The results show that GPNN significantly improves the classification performance of state-of-the-art methods. In addition, analyses also reveal the privilege of the proposed GPNN in filtering out irrelevant neighbors and reducing over-smoothing.

Motivation & Objective

• Address the limitations of existing GNNs in heterophilic graphs, where connected nodes often have dissimilar features or labels.
• Overcome the over-smoothing problem caused by indiscriminate neighbor aggregation in deep GNNs.
• Improve node representation learning by enabling non-local, selective aggregation of distant but semantically relevant nodes.
• Jointly optimize node ranking with downstream tasks to avoid sub-optimal solutions from non-differentiable ranking procedures.
• Develop a scalable and effective framework that outperforms existing methods on real-world heterophilic graph datasets.

Proposed method

• Sample a sequence of nodes from both local and multi-hop neighborhoods around each central node.
• Employ a pointer network to rank these sampled nodes based on their relevance to the central node, producing an ordered sequence of the most informative neighbors.
• Apply a 1D convolutional layer to the ranked sequence to extract high-level structural and semantic features.
• Integrate the pointer network as a differentiable, end-to-end trainable module that is jointly optimized with the rest of the GNN architecture.
• Combine the central node’s ego-features with the contextual features from the ranked neighbor sequence for final node representation.
• Use a differentiable ranking mechanism that allows backpropagation to refine both node selection and representation learning simultaneously.

Experimental results

Research questions

• RQ1 Can a differentiable, end-to-end trainable pointer network effectively identify the most relevant neighbors in heterophilic graphs, even when they are distant?
• RQ2 Does selective, non-local aggregation using a pointer network reduce over-smoothing compared to standard GNNs with deep stacks?
• RQ3 How does the homophily ratio of selected neighbors compare between GPNN and non-differentiable ranking methods like Node2Seq?
• RQ4 To what extent does GPNN improve node classification accuracy on real-world heterophilic graph benchmarks compared to state-of-the-art GNNs?
• RQ5 Can the joint optimization of node ranking and representation learning lead to better performance than separate or heuristic ranking strategies?

Key findings

• GPNN achieves an average 6.3% improvement in node classification accuracy over the best existing state-of-the-art methods across six public heterophilic graph datasets.
• On the Chameleon dataset, GPNN improves performance by 1.9% over the second-best method, and by 3.0% on Cornell, demonstrating consistent gains across diverse benchmarks.
• The homophily ratio of the five most relevant neighbors selected by GPNN is 46% higher than that of neighbors ranked by Node2Seq’s attention-based method, indicating superior neighbor filtering.
• GPNN exhibits significantly reduced over-smoothing: test accuracy drops only 6% when increasing layers from 2 to 5, compared to 18% for GCN and 33% for GAT on the Chameleon dataset.
• The pointer network’s joint optimization with the GNN leads to better performance than non-differentiable ranking methods, as shown by the higher homophily and accuracy gains.
• GPNN consistently outperforms H2GCN and Node2Seq, which do not explicitly distinguish between relevant and irrelevant multi-hop neighbors, confirming the benefit of selective, differentiable neighbor selection.