[Paper Review] Design Space for Graph Neural Networks

Jiaxuan You, Rex Ying+1 more|arXiv (Cornell University)|Nov 17, 2020

Advanced Graph Neural NetworksComputer Science55 references159 citations

TL;DR

This paper defines a general GNN design space with 315,000 designs across 32 tasks, introduces a task similarity space to guide design transfer, and provides a scalable evaluation method plus the GraphGym platform to enable reproducible GNN research.

ABSTRACT

The rapid evolution of Graph Neural Networks (GNNs) has led to a growing number of new architectures as well as novel applications. However, current research focuses on proposing and evaluating specific architectural designs of GNNs, as opposed to studying the more general design space of GNNs that consists of a Cartesian product of different design dimensions, such as the number of layers or the type of the aggregation function. Additionally, GNN designs are often specialized to a single task, yet few efforts have been made to understand how to quickly find the best GNN design for a novel task or a novel dataset. Here we define and systematically study the architectural design space for GNNs which consists of 315,000 different designs over 32 different predictive tasks. Our approach features three key innovations: (1) A general GNN design space; (2) a GNN task space with a similarity metric, so that for a given novel task/dataset, we can quickly identify/transfer the best performing architecture; (3) an efficient and effective design space evaluation method which allows insights to be distilled from a huge number of model-task combinations. Our key results include: (1) A comprehensive set of guidelines for designing well-performing GNNs; (2) while best GNN designs for different tasks vary significantly, the GNN task space allows for transferring the best designs across different tasks; (3) models discovered using our design space achieve state-of-the-art performance. Overall, our work offers a principled and scalable approach to transition from studying individual GNN designs for specific tasks, to systematically studying the GNN design space and the task space. Finally, we release GraphGym, a powerful platform for exploring different GNN designs and tasks. GraphGym features modularized GNN implementation, standardized GNN evaluation, and reproducible and scalable experiment management.

Motivation & Objective

• Motivate moving beyond single-architecture evaluation to studying the full GNN design space.
• Define a general GNN design space that covers intra-layer, inter-layer, and learning configuration dimensions.
• Propose a GNN task space with a task similarity metric to enable fast design transfer across tasks.
• Develop an efficient design space evaluation method to distill insights from millions of model-task combinations.
• Release GraphGym as a modular, standardized platform for exploring GNN designs and tasks.

Proposed method

• Define 12 design dimensions spanning intra-layer, inter-layer, and training configurations to form a 315,000-design GNN space.
• Develop a task space by measuring task similarity via Kendall rank correlation of anchor-model performance across tasks.
• Use a controlled random search to evaluate designs across many model-task combinations within a fixed computational budget.
• Identify guideline design choices through ranking analyses and statistical testing across 96 experiments per design dimension.
• Condense the design space for practical transfer experiments and validate by transferring designs across tasks.
• Release GraphGym with modular GNN implementations and standardized evaluation to enable reproducible experiments.

Experimental results

Research questions

• RQ1 Can a general GNN design space reveal robust design principles beyond individual architectures like GCN or GAT?
• RQ2 How transferable are top-performing GNN designs across similar tasks, as defined by the proposed task similarity metric?
• RQ3 What guidelines emerge for effective GNN design across intra-layer, inter-layer, and training configurations?

Key findings

• Adding BatchNorm generally helps GNN training.
• PReLU emerges as a strong activation choice for GNNs.
• Sum aggregation is empirically very expressive across tasks.
• Skip-Cat connections often outperform Skip-Sum in inter-layer designs.
• Batch size 32 and learning rate 0.01 are favorable, with Adam outperforming SGD in many cases.
• The task space enables transferring best designs across similar tasks, with high correlations in transfer performance.
• Condensed design space can achieve state-of-the-art performance on challenging new tasks (e.g., ogbg-molhiv).