[Paper Review] Generalized Laplacian Positional Encoding for Graph Representation Learning

Sohir Maskey, Ali Parviz+4 more|arXiv (Cornell University)|Oct 28, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper introduces a generalized Laplacian positional encoding (p-PE) that extends traditional Laplacian positional encoding by replacing the $L^2$-norm with $p$-norms in the embedding optimization problem. The method computes node embeddings via local minimization, capturing diverse graph structures—such as rings in molecules—especially when $p \to 1$, though experiments show $p$-PE generally underperforms standard $2$-PE in node classification and graph regression tasks.

ABSTRACT

Graph neural networks (GNNs) are the primary tool for processing graph-structured data. Unfortunately, the most commonly used GNNs, called Message Passing Neural Networks (MPNNs) suffer from several fundamental limitations. To overcome these limitations, recent works have adapted the idea of positional encodings to graph data. This paper draws inspiration from the recent success of Laplacian-based positional encoding and defines a novel family of positional encoding schemes for graphs. We accomplish this by generalizing the optimization problem that defines the Laplace embedding to more general dissimilarity functions rather than the 2-norm used in the original formulation. This family of positional encodings is then instantiated by considering p-norms. We discuss a method for calculating these positional encoding schemes, implement it in PyTorch and demonstrate how the resulting positional encoding captures different properties of the graph. Furthermore, we demonstrate that this novel family of positional encodings can improve the expressive power of MPNNs. Lastly, we present preliminary experimental results.

Motivation & Objective

• Address the limitations of Message Passing Neural Networks (MPNNs), such as bounded expressive power and poor performance on link prediction.
• Overcome the arbitrary use of the $L^2$-norm in Laplacian-based positional encoding (LPE) by generalizing the optimization problem to arbitrary dissimilarity functions.
• Develop a novel family of positional encodings using $p$-norms to better capture diverse graph properties, including clustering and ring structures.
• Demonstrate that $p$-PE can enhance MPNN expressivity, particularly when using $p$-eigenvectors, and explore its theoretical and empirical potential.
• Identify challenges in optimizing $p$-PE efficiently and propose future directions for scalable and effective $p$-norm-based encodings.

Proposed method

• Generalize the standard Laplacian embedding problem by replacing the $L^2$-norm with a general dissimilarity function $d(\cdot, \cdot)$ in the objective: $\min_X \sum_{i,j} a_{ij} d(X_{i,:}, X_{j,:})$ subject to constraints $C(X) = 0$.
• Instantiate the general framework using $p$-norms, defining $p$-PE as the solution to a non-convex optimization problem that minimizes $p$-norm-based pairwise dissimilarities under orthogonality constraints.
• Implement a local minimization scheme to compute $p$-PE embeddings, avoiding full eigendecomposition and enabling scalability for small $k$.
• Apply sign-flipping and SignNet (SN) to handle the sign ambiguity inherent in $p$-PE, similar to standard LPE.
• Use the resulting $p$-PE as initial node features in GNNs like GCN, GIN, and Transformer-based models for downstream tasks.
• Theoretical analysis shows that MPNNs augmented with two or more $p$-eigenvectors are strictly more expressive than the 1-WL test for any $p \in (1, \infty)$.

Experimental results

Research questions

• RQ1 Can replacing the $L^2$-norm in Laplacian positional encoding with $p$-norms yield more expressive and structurally meaningful node embeddings?
• RQ2 How do $p$-PE embeddings differ in their ability to capture graph properties such as rings, clusters, or shortest paths compared to standard $2$-PE?
• RQ3 What is the empirical performance of $p$-PE in comparison to $2$-PE across node classification and graph regression benchmarks?
• RQ4 Can $p$-PE improve the expressive power of MPNNs beyond the 1-WL test, particularly when using $p$-eigenvectors?
• RQ5 What are the main computational and optimization challenges in scaling $p$-PE to larger $k$ or larger graphs?

Key findings

• The proposed $p$-PE framework generalizes Laplacian positional encoding by replacing the $L^2$-norm with $p$-norms in the embedding optimization, enabling diverse structural inductive biases.
• When $p \to 1$, $p$-PE behaves like a soft clustering function and effectively identifies meaningful graph structures such as rings in molecular graphs, as demonstrated in visualization on ZINC molecules.
• Despite theoretical advantages, $p$-PE underperforms standard $2$-PE in node classification on Cora and Citeseer, with GCN+1.2-PE achieving 83.4% and 75.2% accuracy, respectively, compared to 83.6% and 74.7% for GCN+No PE.
• In graph regression on ZINC-12K, $2$-PE-Norm. achieved the lowest test MAE of $0.168 \pm 0.02$ with GINE, while $1.2$-PE and $1.6$-PE achieved $0.180 \pm 0.01$ and $0.171 \pm 0.00$, respectively, showing consistent but slightly worse performance than $2$-PE.
• Transformer models with $2$-PE-Norm. achieved the best performance on ZINC ($0.121 \pm 0.00$ MAE), while $1.2$-PE and $1.6$-PE yielded $0.133 \pm 0.01$ and $0.134 \pm 0.01$, indicating that $p$-PE does not outperform $2$-PE in this setting.
• Theoretical results confirm that MPNNs augmented with at least two $p$-eigenvectors are strictly more expressive than the 1-WL test for any $p \in (1, \infty)$, suggesting potential for improved expressivity in theory, though empirical gains remain limited in current experiments.