[Paper Review] Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels

Binxin Ru, Xingchen Wan+2 more|arXiv (Cornell University)|Jun 13, 2020

Machine Learning and Data ClassificationComputer Science82 references20 citations

TL;DR

This paper proposes NAS-BOWL, a Bayesian optimization framework for neural architecture search that uses a Gaussian process surrogate with Weisfeiler-Lehman graph kernels to model neural architecture search spaces. By capturing topological structure and enabling interpretable feature extraction, NAS-BOWL achieves state-of-the-art performance with high data efficiency and provides insights into performance-driving architectural motifs.

ABSTRACT

Current neural architecture search (NAS) strategies focus only on finding a single, good, architecture. They offer little insight into why a specific network is performing well, or how we should modify the architecture if we want further improvements. We propose a Bayesian optimisation (BO) approach for NAS that combines the Weisfeiler-Lehman graph kernel with a Gaussian process surrogate. Our method optimises the architecture in a highly data-efficient manner: it is capable of capturing the topological structures of the architectures and is scalable to large graphs, thus making the high-dimensional and graph-like search spaces amenable to BO. More importantly, our method affords interpretability by discovering useful network features and their corresponding impact on the network performance. Indeed, we demonstrate empirically that our surrogate model is capable of identifying useful motifs which can guide the generation of new architectures. We finally show that our method outperforms existing NAS approaches to achieve the state of the art on both closed- and open-domain search spaces.

Motivation & Objective

• To address the black-box nature of existing NAS methods by introducing interpretability into architecture search.
• To enable data-efficient optimization in high-dimensional, graph-like NAS search spaces.
• To identify meaningful architectural motifs that explain performance and guide future design.
• To outperform existing BO-based NAS methods in both closed- and open-domain search spaces.

Proposed method

• The method uses a Gaussian process surrogate (GPWL) combined with the Weisfeiler-Lehman subtree kernel to model the topology of neural architecture graphs.
• Architecture representations are encoded as node-attributed directed acyclic graphs (DAGs), preserving topological structure for kernel computation.
• The GPWL surrogate enables data-efficient learning with uncertainty quantification and automatic hyperparameter handling.
• Interpretable features are extracted via the WL kernel, and their impact on performance is analyzed using surrogate gradients.
• A mutation-based acquisition strategy is employed to generate new candidate architectures in each BO iteration.
• The framework supports both closed-domain (NAS-Bench) and open-domain (DARTS-style) search spaces.

Experimental results

Research questions

• RQ1 Can a kernel-based GP surrogate effectively model the topological structure of neural architecture search spaces?
• RQ2 Can the WL kernel extract interpretable architectural motifs that correlate with performance?
• RQ3 Does the GPWL surrogate enable data-efficient and high-performing NAS compared to existing methods?
• RQ4 Can the identified motifs be used to guide transfer learning and warm-start search on new tasks?

Key findings

• NAS-BOWL achieves state-of-the-art performance on both NAS-Bench 101 and 201 datasets, outperforming existing BO-based NAS methods with fewer queries.
• The method achieves superior data efficiency, converging to optimal performance with significantly fewer architecture evaluations than competing approaches.
• The WL kernel's multi-scale feature extraction is crucial: using only node features (VH) leads to substantial performance drops, highlighting the importance of topological structure.
• Interpretable features extracted via the WL kernel correspond to meaningful architectural motifs, which are validated as useful for transfer learning.
• The choice of acquisition function (UCB vs. expected improvement) has minimal impact on performance, suggesting robustness to acquisition strategy.
• The use of a mutation algorithm significantly improves search performance, demonstrating its effectiveness in exploring the search space.