[Paper Review] Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning

Pan Li, Yanbang Wang+2 more|arXiv (Cornell University)|Aug 31, 2020

Advanced Graph Neural NetworksComputer Science68 references98 citations

TL;DR

Introduces Distance Encoding (DE) to boost GNN expressiveness beyond 1-WL by using node-set distances; provides DE-GNN and DEA-GNN frameworks with theoretical guarantees and empirical improvements on node-set tasks.

ABSTRACT

Learning representations of sets of nodes in a graph is crucial for applications ranging from node-role discovery to link prediction and molecule classification. Graph Neural Networks (GNNs) have achieved great success in graph representation learning. However, expressive power of GNNs is limited by the 1-Weisfeiler-Lehman (WL) test and thus GNNs generate identical representations for graph substructures that may in fact be very different. More powerful GNNs, proposed recently by mimicking higher-order-WL tests, only focus on representing entire graphs and they are computationally inefficient as they cannot utilize sparsity of the underlying graph. Here we propose and mathematically analyze a general class of structure-related features, termed Distance Encoding (DE). DE assists GNNs in representing any set of nodes, while providing strictly more expressive power than the 1-WL test. DE captures the distance between the node set whose representation is to be learned and each node in the graph. To capture the distance DE can apply various graph-distance measures such as shortest path distance or generalized PageRank scores. We propose two ways for GNNs to use DEs (1) as extra node features, and (2) as controllers of message aggregation in GNNs. Both approaches can utilize the sparse structure of the underlying graph, which leads to computational efficiency and scalability. We also prove that DE can distinguish node sets embedded in almost all regular graphs where traditional GNNs always fail. We evaluate DE on three tasks over six real networks: structural role prediction, link prediction, and triangle prediction. Results show that our models outperform GNNs without DE by up-to 15\% in accuracy and AUROC. Furthermore, our models also significantly outperform other state-of-the-art methods especially designed for the above tasks.

Motivation & Objective

• Address the limited expressive power of standard WLGNNs bound by the 1-WL test.
• Introduce Distance Encoding (DE) to encode distances from a target node set to all graph nodes.
• Propose two DE-enabled frameworks (DE-GNN and DEA-GNN) that integrate DEs as features and as controllers of message aggregation.
• Provide theoretical guarantees showing DE-based models distinguish most non-isomorphic structures in sparse regular graphs.
• Demonstrate empirical gains on node-structural roles, link prediction, and triangle prediction tasks across real networks.

Proposed method

• Define Distance Encoding (DE) as a permutation-invariant function that maps each graph node to a vector of landing probabilities from the target node set, using measures such as shortest-path distance (SPD) and generalized PageRank scores.
• Use DE either as (i) extra node features in a DE-GNN, where h_v^(0) includes DE(v|S); or (ii) as controllers in aggregation in a DEA-GNN, where DE(u|v) guides neighbor aggregation.
• Provide that DE can be implemented via a simple aggregation, e.g., zeta(u|S) = AGG{ zeta(u|v) : v in S } with zeta(u|v) = f3(l_uv) and l_uv derived from powers of the random-walk matrix W = AD^{-1} (e.g., SPD or GPR).
• Theoretical results show: (a) DE-GNN/DEA-GNN can distinguish equal-sized node sets embedded in almost all sparse r-regular graphs with high probability in O(log n) depth, (b) limitations on DRGs where DE-1 is insufficient but DE-2 can help, (c) DE-1 has limitations on distance-regular graphs, and (d) higher-order DE (p>=2) can overcome some DRG limitations.

Experimental results

Research questions

• RQ1 Can Distance Encoding (DE) provide strictly more expressive power than 1-WL for structural representations of node sets?
• RQ2 Do DE-GNN and DEA-GNN frameworks provably distinguish non-isomorphic node sets in sparse regular graphs where standard WLGNNs fail?
• RQ3 What are the theoretical limitations of DE-based approaches, particularly on distance-regular graphs?
• RQ4 Do DE-GNN and DEA-GNN yield empirical improvements on node-structural roles, link prediction, and triangle prediction tasks on real networks?

Key findings

• DE-equipped GNNs can distinguish non-isomorphic equal-sized node sets embedded in almost all sparse r-regular graphs with high probability, using SPD as the DE component.
• DE-GNN and DEA-GNN outperform standard WLGNN baselines by up to 15% in average accuracy or AUC across node-structural-role classification, link prediction, and triangle prediction tasks.
• DE-1 has limitations on distance-regular graphs, but DE-2 (using node-pair level encoding) can distinguish certain DRGs where DE-1 cannot.
• DE-based methods remain bounded by the 2-WL test in some DRG settings unless higher-order DE representations are employed.
• Empirical evaluations on six real networks show DE-GNN/DEA-GNN outperform baselines including Ring-GNN/PPGN and SEAL for the considered tasks.