[Paper Review] Labeling Trick: A Theory of Using Graph Neural Networks for Multi-Node Representation Learning

Muhan Zhang, Pan Li+3 more|arXiv (Cornell University)|Oct 30, 2020

Advanced Graph Neural Networks45 references59 citations

TL;DR

The paper introduces labeling trick to enable GNNs to learn expressive joint representations for node sets, proving theoretical guarantees (Theorem 1 and related results) and showing SEAL’s effectiveness arises from labeling. It unifies prior node-labeling approaches and analyzes practicality via local isomorphism and experiments on OGBL datasets.

ABSTRACT

In this paper, we provide a theory of using graph neural networks (GNNs) for multi-node representation learning (where we are interested in learning a representation for a set of more than one node, such as link). We know that GNN is designed to learn single-node representations. When we want to learn a node set representation involving multiple nodes, a common practice in previous works is to directly aggregate the single-node representations obtained by a GNN into a joint node set representation. In this paper, we show a fundamental constraint of such an approach, namely the inability to capture the dependence between nodes in the node set, and argue that directly aggregating individual node representations does not lead to an effective joint representation for multiple nodes. Then, we notice that a few previous successful works for multi-node representation learning, including SEAL, Distance Encoding, and ID-GNN, all used node labeling. These methods first label nodes in the graph according to their relationships with the target node set before applying a GNN. Then, the node representations obtained in the labeled graph are aggregated into a node set representation. By investigating their inner mechanisms, we unify these node labeling techniques into a single and most general form -- labeling trick. We prove that with labeling trick a sufficiently expressive GNN learns the most expressive node set representations, thus in principle solves any joint learning tasks over node sets. Experiments on one important two-node representation learning task, link prediction, verified our theory. Our work explains the superior performance of previous node-labeling-based methods, and establishes a theoretical foundation of using GNNs for multi-node representation learning.

Motivation & Objective

• Identify the fundamental limitation of directly aggregating single-node GNN embeddings for multi-node prediction tasks.
• Develop a unified theory (labeling trick) for learning node-set representations with GNNs.
• Establish conditions under which labeling trick yields the most expressive structural representations (Theorem 1).
• Analyze connections to existing labeling schemes (SEAL, DE, DRNL) and extend the theory to practical settings such as local isomorphism.
• Provide empirical validation on link prediction tasks to corroborate the theory.

Proposed method

• Define labeling trick as stacking a labeling tensor with the graph to form a labeled graph, ensuring target-nodes-distinguishing and permutation equivariance.
• Introduce zero-one labeling as a simple, valid labeling trick to distinguish target node sets.
• Prove Theorem 1: with a node-most-expressive GNN and injective set aggregation, GNN on the labeled graph yields a structural representation of the node set.
• Prove Theorem 2: in non-attributed graphs, 1-WL GNNs with labeling trick can distinguish many non-isomorphic links that plain 1-WL cannot.
• Demonstrate SEAL uses a labeling trick via DRNL labeling of enclosing subgraphs and relate it to the theory (Theorem 3).
• Discuss extensions to Distance Encoding (DE), heterogeneous graphs, and local isomorphism as practical refinements.

Experimental results

Research questions

• RQ1 Can directly aggregating single-node GNN embeddings produce a structural representation for node sets?
• RQ2 Does labeling trick enable a node-wise GNN to learn the most expressive joint representations for node sets?
• RQ3 How do SEAL and other labeling-based methods fit within the labeling trick framework?
• RQ4 What is the impact of local isomorphism vs exact isomorphism in learning node-set representations?
• RQ5 How do labeling tricks perform in practice on link prediction tasks and heterogeneous graphs?

Key findings

• Directly aggregating node representations with GAE cannot learn structural link representations, even with highly expressive node embeddings.
• Labeling trick enables a node-most-expressive GNN to learn structural representations for node sets via labeled graphs and injective set aggregation (Theorem 1).
• In non-attributed graphs, there exist many link pairs indistinguishable by 1-WL GNN alone but distinguishable with labeling trick (Theorem 2).
• SEAL’s use of DRNL labeling is an instance of labeling trick, explaining its empirical advantage over GAE (Theorem 3).
• The framework generalizes to DE, heterogeneous graphs, and practical notions such as local isomorphism, providing a broad theoretical foundation for multi-node learning with GNNs.