[论文解读] Auto-GNN: Neural Architecture Search of Graph Neural Networks

Kaixiong Zhou, Qingquan Song+2 more|arXiv (Cornell University)|Sep 7, 2019

Advanced Graph Neural Networks参考文献 44被引用 64

一句话总结

本文提出 AGNN，一种面向图神经网络的强化学习驱动 NAS 框架，结合新颖的保守探索和受约束的参数共享以高效发现适用于节点分类的高性能 GNN 架构。

ABSTRACT

Graph neural networks (GNN) has been successfully applied to operate on the graph-structured data. Given a specific scenario, rich human expertise and tremendous laborious trials are usually required to identify a suitable GNN architecture. It is because the performance of a GNN architecture is significantly affected by the choice of graph convolution components, such as aggregate function and hidden dimension. Neural architecture search (NAS) has shown its potential in discovering effective deep architectures for learning tasks in image and language modeling. However, existing NAS algorithms cannot be directly applied to the GNN search problem. First, the search space of GNN is different from the ones in existing NAS work. Second, the representation learning capacity of GNN architecture changes obviously with slight architecture modifications. It affects the search efficiency of traditional search methods. Third, widely used techniques in NAS such as parameter sharing might become unstable in GNN. To bridge the gap, we propose the automated graph neural networks (AGNN) framework, which aims to find an optimal GNN architecture within a predefined search space. A reinforcement learning based controller is designed to greedily validate architectures via small steps. AGNN has a novel parameter sharing strategy that enables homogeneous architectures to share parameters, based on a carefully-designed homogeneity definition. Experiments on real-world benchmark datasets demonstrate that the GNN architecture identified by AGNN achieves the best performance, comparing with existing handcrafted models and tradistional search methods.

研究动机与目标

• 基于信息传递卷积定义图神经架构的搜索空间。
• 开发一个具有因果性的强化学习控制器来学习架构修改。
• 引入受约束的参数共享策略，以在异构 GNNs 间稳定训练。
• 在节点分类任务上实证验证 AGNN 找到的架构优于手工模型和其他 NAS 方法。

提出的方法

• 为每个 GNN 层定义六类动作空间：隐藏维度、注意力函数、注意力头数、聚合函数、组合函数和激活函数。
• 使用基于强化学习的控制器，对每个动作类别设独立的 RNN 编码器，贪婪地修改架构。
• 实现 reinforced conservative neural architecture search (RCNAS)，具有保守探测器、引导性的架构修改器和策略梯度训练器。
• 提出受约束的参数共享，使权重只在输入/输出形状匹配、注意力/激活函数相同且不共享 BN/跳跃连接组件的架构之间进行传递。
• 用共享权重训练子架构若干轮，以在探索与稳定性之间取得平衡。

实验结果

研究问题

• RQ1 如何为 GNN 架构定义一个高效的搜索空间并有效地进行探索？
• RQ2 如何约束参数共享以在异质 GNN 架构之间稳定训练？
• RQ3 AGNN 是否能够找到在节点分类任务上超越手工模型和其他 NAS 方法的架构？
• RQ4 对比有限修改与全量架构修改，对搜索效率有何影响？

主要发现

Baseline Class	Model	#Layers	Params	Cora	Citeseer	Pubmed
Chebyshev	2-layer	2	0.09 M	81.2%	69.8%	74.4%
GCN	2-layer	2	0.02 M	81.5%	70.3%	79.0%
GAT	2-layer	2	0.09 M	83.0±0.7%	72.5±0.7%	79.0±0.3%
LGCN	3–4-layer	3–4	0.06 M	83.3±0.5%	73.0±0.6%	79.5±0.2%
GraphNAS-w/o share	2-layer	2	0.09 M	82.7±0.4%	73.5±1.0%	78.8±0.5%
GraphNAS-with share	2-layer	2	1.91 M	72.4±1.3%	-	-
Random-w/o share	2-layer	2	0.95 M	72.9±0.2%	-	-
Random-with share	2-layer	2	0.95 M	69.9±1.7%	-	-
AGNN-w/o share	2-layer	2	0.71 M	83.8%	73.8%	79.7%
AGNN-with share	2-layer	2	0.03 M	82.7%	72.7%	79.0%

• AGNN 发现的架构在传导式节点分类基准测试上优于手工模型和 NAS 基线。例如，AGNN-w/o share 在 Cora、Citeseer、Pubmed 的表现分别为 83.6%、73.8%、79.7%，优于若干基线。
• 在传导式任务上，AGNN-with share 也表现出竞争力（如 82.7% Cora、72.7% Citeseer、79.0% Pubmed），且使用的参数量少于某些 NAS 基线。
• 在 PPI 的归纳学习中，AGNN-w/o share 达到 0.992±0.001 的 F1 分数，AGNN-with share 达到 0.991±0.001，优于随机和若干手工基线。
• 搜索过程表明 AGNN 能比 GraphNAS 和随机搜索更快收敛到顶尖架构，参数共享在实践中降低了计算时间。
• 受约束的参数共享通过仅在具有匹配层形状和相同注意力/激活配置的架构之间强制共享权重，从而稳定异构 GNN 架构的训练。