[Paper Review] Graph Transformer Networks

Seongjun Yun, Minbyul Jeong+3 more|arXiv (Cornell University)|Nov 6, 2019

Advanced Graph Neural Networks514 citations

TL;DR

GTNs learn to transform a heterogeneous graph into multiple meta-path graphs and perform end-to-end node classification without manually defined meta-paths, achieving state-of-the-art results on benchmarks.

ABSTRACT

Graph neural networks (GNNs) have been widely used in representation learning on graphs and achieved state-of-the-art performance in tasks such as node classification and link prediction. However, most existing GNNs are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a misspecified graph or a heterogeneous graph that consists of various types of nodes and edges. In this paper, we propose Graph Transformer Networks (GTNs) that are capable of generating new graph structures, which involve identifying useful connections between unconnected nodes on the original graph, while learning effective node representation on the new graphs in an end-to-end fashion. Graph Transformer layer, a core layer of GTNs, learns a soft selection of edge types and composite relations for generating useful multi-hop connections so-called meta-paths. Our experiments show that GTNs learn new graph structures, based on data and tasks without domain knowledge, and yield powerful node representation via convolution on the new graphs. Without domain-specific graph preprocessing, GTNs achieved the best performance in all three benchmark node classification tasks against the state-of-the-art methods that require pre-defined meta-paths from domain knowledge.

Motivation & Objective

• Motivate learning node representations on heterogeneous graphs beyond fixed, homogeneous graphs.
• Develop a framework that automatically discovers useful meta-paths and multi-hop connections for each task.
• Enable end-to-end training where graph structure learning and node representation learning are integrated.
• Provide interpretability by identifying which meta-paths (edge-types) contribute to predictions.

Proposed method

• Represent heterogeneous graphs using multiple adjacency matrices for different edge types.
• Use 1x1 convolutions with softmax to softly weight and select candidate edge-types.
• Form new meta-path graphs by multiplying selected adjacency matrices (Q1 Q2) and normalize as A(l) = D^{-1} Q1 Q2.
• Include the identity matrix in the candidate set to allow learning variable path lengths andOriginal edges.
• Apply GCN on each learned meta-path graph and concatenate multi-channel representations for final prediction.
• Train with standard cross-entropy loss on labeled nodes.

Experimental results

Research questions

• RQ1 RQ1: Are the new graph structures generated by GTN effective for learning node representations?
• RQ2 RQ2: Can GTN adaptively produce a variable length of meta-paths depending on datasets?
• RQ3 RQ3: How can we interpret the importance of each meta-path from the adjacency matrices generated by GTNs?
• RQ4 (Extra) Do GTNs outperform methods that require predefined meta-paths on heterogeneous graph node classification?

Key findings

• GTN achieves the highest performance on all three datasets compared to both random-walk baselines and other GNNs.
• GAT typically outperforms GCN due to learned neighbor weights, while HAN underperforms relative to GAT in these experiments.
• GTN without identity (GTN_-I) performs worse than the full GTN, indicating the benefit of learning shorter meta-paths.
• GTN learns meta-paths that align with domain-defined meta-paths for target nodes and also discovers novel, informative meta-paths.
• The model can adapt meta-path length depending on the dataset, placing higher attention on identity when shorter paths are beneficial (as observed in IMDB).
• An ablation shows GTNs act as an ensemble of GCNs on multiple learned meta-path graphs, improving performance with end-to-end learning.