[Paper Review] Graph Positional and Structural Encoder

Semih Cantürk, Renming Liu+5 more|arXiv (Cornell University)|Jul 14, 2023

Advanced Graph Neural NetworksComputer Science3 citations

TL;DR

This paper introduces the Graph Positional and Structural Encoder (GPSE), a learnable graph encoder that jointly captures positional and structural information from graph topology, replacing hand-crafted positional and structural encodings (PSEs) in GNNs. GPSE is pre-trained on diverse graph datasets using random node features to ensure expressivity, achieving state-of-the-art performance on ZINC and Peptides-struct while generalizing effectively across datasets with different sizes, connectivities, and modalities.

ABSTRACT

Positional and structural encodings (PSE) enable better identifiability of nodes within a graph, rendering them essential tools for empowering modern GNNs, and in particular graph Transformers. However, designing PSEs that work optimally for all graph prediction tasks is a challenging and unsolved problem. Here, we present the Graph Positional and Structural Encoder (GPSE), the first-ever graph encoder designed to capture rich PSE representations for augmenting any GNN. GPSE learns an efficient common latent representation for multiple PSEs, and is highly transferable: The encoder trained on a particular graph dataset can be used effectively on datasets drawn from markedly different distributions and modalities. We show that across a wide range of benchmarks, GPSE-enhanced models can significantly outperform those that employ explicitly computed PSEs, and at least match their performance in others. Our results pave the way for the development of foundational pre-trained graph encoders for extracting positional and structural information, and highlight their potential as a more powerful and efficient alternative to explicitly computed PSEs and existing self-supervised pre-training approaches. Our framework and pre-trained models are publicly available at https://github.com/G-Taxonomy-Workgroup/GPSE. For convenience, GPSE has also been integrated into the PyG library to facilitate downstream applications.

Motivation & Objective

• To address the lack of a unified, transferable method for learning positional and structural encodings (PSEs) in graph neural networks (GNNs), especially for graph Transformers.
• To overcome the limitations of hand-crafted PSEs, which vary in effectiveness across tasks and lack generalization.
• To develop a pre-trained graph encoder that learns rich, shared representations of positional and structural information across diverse graph distributions.
• To enable transfer learning of PSE representations across datasets with different sizes, connectivity patterns, and modalities.
• To provide a viable alternative to both explicitly computed PSEs and self-supervised pre-training methods in graph representation learning.

Proposed method

• GPSE is a learnable graph encoder trained to predict multiple types of PSEs—such as Laplacian eigenvectors, random walk encodings, and heat kernel features—simultaneously from graph structure.
• The model uses random node features (standard normal distribution) during pre-training to break symmetry and ensure that non-isomorphic graphs are distinguishable, even under 1-WL limitations.
• The encoder is trained via a multi-task objective combining mean absolute error (MAE) and cosine loss across multiple PSE types to learn a unified latent representation.
• The pre-trained GPSE encoder is fine-tuned or directly applied as node features in downstream GNNs, replacing traditional PSEs or raw node features.
• The architecture is compatible with both message-passing GNNs (e.g., GIN, GatedGCN) and graph Transformers, enabling broad applicability.
• Transferability is evaluated by applying the same pre-trained GPSE encoder to datasets with different graph statistics and modalities, including molecular, social, and synthetic graphs.

Figure 1 : Overview of Graph Positional and Structural Encoder ( GPSE ) training and application.

Experimental results

Research questions

• RQ1 Can a single learnable encoder effectively capture diverse positional and structural encoding patterns from graph topology?
• RQ2 Does GPSE generalize across datasets with different sizes, connectivity patterns, and data modalities?
• RQ3 Can GPSE improve downstream GNN performance compared to hand-crafted PSEs and self-supervised pre-training?
• RQ4 How critical is the use of random node features in enabling GPSE to distinguish non-isomorphic graphs?
• RQ5 To what extent can GPSE replace explicitly computed PSEs without performance degradation?

Key findings

• GPSE achieves state-of-the-art performance on the ZINC and Peptides-struct benchmarks, outperforming models using traditional PSEs.
• On the PCQM4Mv2 dataset, GPSE achieves a test MAE of 0.1196 ± 0.0004 when pre-trained on MolPCBA and fine-tuned, demonstrating strong transferability.
• The inclusion of random node features is essential: GPSE fails to distinguish non-isomorphic graphs (e.g., HEXAGON vs. PENTAGON) when node features are constant, but succeeds with random features.
• Fine-tuning GPSE on downstream datasets consistently improves performance, with MAE decreasing from 0.0707 to 0.0685 on GEOM and from 0.0667 to 0.0643 on ChEMBL.
• GPSE pre-training on large datasets like MolPCBA (323,555 graphs) leads to strong generalization, with test loss decreasing from 0.06939 (5%) to 0.01219 (80%) as training size increases.
• Ablation studies show that excluding any single PSE task during pre-training has minimal negative impact, indicating robustness and complementary learning across PSE types.

Figure 2 : Virtual node, convolution type, and layers ablation using 5% MolPCBA for training GPSE .