[Paper Review] Representation Learning on Graphs with Jumping Knowledge Networks

Keyulu Xu, Chengtao Li+4 more|arXiv (Cornell University)|Jun 9, 2018

Advanced Graph Neural Networks21 references726 citations

TL;DR

JK-Nets adaptively combine neighborhood ranges for each node, improving performance over standard GCN/GAT and consistently boosting base models like GraphSAGE and GAT across diverse graphs.

ABSTRACT

Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.

Motivation & Objective

• Motivate and analyze limitations of fixed-range neighborhood aggregation in graph neural networks.
• Propose JK-Nets to adaptively select information from multiple neighborhood ranges per node.
• Demonstrate that JK-Nets improve performance when combined with baseline models (GCN, GraphSAGE, GAT).
• Empirically evaluate on multiple datasets (Citeseer, Cora, Reddit, PPI) to show robustness across graph structures.

Proposed method

• Introduce Jumping Knowledge (JK) framework that ends with a last-layer aggregation selectively combining representations from multiple layers.
• Provide three JK aggregation options: Concat, MaxPooling, and LSTM-attention to fuse multi-layer representations.
• Establish theoretical connections between influence distributions of aggregations and random walks; prove thatJK-Net with max-pooling induces a mixture of k-step random-walk distributions.
• Show compatibility and improvements when JK is plugged into GCN, GraphSAGE, and GAT architectures.

Experimental results

Research questions

• RQ1 Can adaptive, node-specific selection of neighborhood ranges beat fixed-range aggregations in graph neural networks?
• RQ2 How do JK-Nets relate to random walk interpretations of information spread on graphs?
• RQ3 Do JK-Nets consistently improve performance when combined with diverse base models across different graph domains?
• RQ4 What are the empirical gains of JK-Nets on citation, social/reddit, and biological networks?

Key findings

• JK-Nets outperform GCN and GAT baselines on Citeseer and Cora datasets across tested layer configurations.
• On Citeseer, JK-Nets with simple aggregations achieve up to 0.78 accuracy with 1 layer and competitive performance with 6 layers (e.g., JK-MaxPool 1: 77.7, JK-Concat 1: 78.3, JK-LSTM 2: 74.7).
• On Cora, JK-Nets achieve best performance with 6 layers (e.g., JK-MaxPool 6: 89.6, JK-Concat 6: 89.1, JK-LSTM 1: 85.8).
• For Reddit, JK-Net with GraphSAGE base and MaxPool+Concat aggregation reaches Micro-F1 up to 0.965, improving over the baseline GraphSAGE.
• On PPI, JK-Nets with LSTM-attention outperform baselines (GAT, GraphSAGE variants) with Micro-F1 up to 0.969–0.976 across configurations.
• JK-LSTM and other adaptive JK variants particularly excel on graphs with diverse subgraph structures (e.g., PPI) where fixed-range methods underperform.