[Paper Review] Search to aggregate neighborhood for graph neural network

Huan Zhao, Quanming Yao+1 more|arXiv (Cornell University)|Apr 14, 2021

Advanced Graph Neural NetworksComputer Science52 references26 citations

TL;DR

SANE automatically designs data-specific GNN architectures via differentiable neural architecture search, yielding more effective and efficient models than baselines across multiple datasets.

ABSTRACT

Recent years have witnessed the popularity and success of graph neural networks (GNN) in various scenarios. To obtain data-specific GNN architectures, researchers turn to neural architecture search (NAS), which has made impressive success in discovering effective architectures in convolutional neural networks. However, it is non-trivial to apply NAS approaches to GNN due to challenges in search space design and the expensive searching cost of existing NAS methods. In this work, to obtain the data-specific GNN architectures and address the computational challenges facing by NAS approaches, we propose a framework, which tries to Search to Aggregate NEighborhood (SANE), to automatically design data-specific GNN architectures. By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods. Experimental results on four tasks and seven real-world datasets demonstrate the superiority of SANE compared to existing GNN models and NAS approaches in terms of effectiveness and efficiency. (Code is available at: https://github.com/AutoML-4Paradigm/SANE).

Motivation & Objective

• Motivate the need for data-specific GNN architectures due to dataset/task heterogeneity.
• Propose a novel expressive search space for GNNs focusing on node and layer aggregators and skip connections.
• Develop a differentiable one-shot NAS method to efficiently search over the proposed space.
• Empirically validate SANE across transductive, inductive, and DB tasks on multiple real-world datasets.

Proposed method

• Define an expressive search space with node aggregators, layer aggregators, and skip connections that can emulate many GNN architectures.
• Construct a supernet (DAG) where each edge has a mixed operation from predefined sets; relax the discrete choice with softmax over candidate operations.
• Formulate a bi-level optimization: architecture parameters α optimized on validation loss, network weights w optimized on training loss.
• Apply one-shot differentiable NAS to update α via gradient-based updates and weights via standard training, then extract the discrete architecture.
• Retain the top-1 architecture from multiple search runs and retrain with tuned hyper-parameters for final evaluation.

Experimental results

Research questions

• RQ1 Can a differentiable NAS framework discover data-specific GNN architectures that outperform manually designed models and RL-based NAS methods?
• RQ2 Does a compact, expressive search space for node and layer aggregators plus skip connections suffice to emulate diverse GNNs?
• RQ3 Is one-shot differentiable NAS efficient enough for practical GNN architecture search on real-world datasets?
• RQ4 How do the discovered architectures perform across transductive, inductive, and database tasks?
• RQ5 What characteristics do the searched architectures reveal about task- and dataset-specific design choices?

Key findings

• SANE consistently outperforms both human-designed GNNs and NAS baselines on transductive and inductive tasks across multiple datasets.
• The proposed expressive search space enables SANE to emulate a wide range of architectures and discover data-specific configurations.
• A differentiable, one-shot NAS approach significantly improves search efficiency over RL-based NAS methods.
• Architectures discovered by SANE are data-dependent and often leverage skip connections and attention-based aggregators to achieve strong performance.
• SANE achieves substantial efficiency gains in search time while delivering state-of-the-art results on diverse tasks.