[Paper Review] Neural Architecture Search with GBDT.

Renqian Luo, Xu Tan+4 more|arXiv (Cornell University)|Jul 9, 2020

Adversarial Robustness in Machine Learning42 references25 citations

TL;DR

This paper proposes using Gradient Boosting Decision Trees (GBDT) as an accuracy predictor in Neural Architecture Search (NAS), leveraging GBDT’s strength in handling tabular architecture data to improve search efficiency and accuracy. By combining GBDT-based search space pruning with GBDT-based architecture prediction, the method achieves state-of-the-art performance, attaining 23.5% top-1 error on ImageNet and 0.18% test regret on NASBench-101.

ABSTRACT

Neural architecture search (NAS) with an accuracy predictor that predicts the accuracy of candidate architectures has drawn increasing interests due to its simplicity and effectiveness. Previous works employ neural network based predictors which unfortunately cannot well exploit the tabular data representations of network architectures. As decision tree-based models can better handle tabular data, in this paper, we propose to leverage gradient boosting decision tree (GBDT) as the predictor for NAS and demonstrate that it can improve the prediction accuracy and help to find better architectures than neural network based predictors. Moreover, considering that a better and compact search space can ease the search process, we propose to prune the search space gradually according to important features derived from GBDT using an interpreting tool named SHAP. In this way, NAS can be performed by first pruning the search space (using GBDT as a pruner) and then searching a neural architecture (using GBDT as a predictor), which is more efficient and effective. Experiments on NASBench-101 and ImageNet demonstrate the effectiveness of GBDT for NAS: (1) NAS with GBDT predictor finds top-10 architecture (among all the architectures in the search space) with $0.18\%$ test regret on NASBench-101, and achieves $24.2\%$ top-1 error rate on ImageNet; and (2) GBDT based search space pruning and neural architecture search further achieves $23.5\%$ top-1 error rate on ImageNet.

Motivation & Objective

• To address the limitation of neural network-based predictors in NAS, which poorly handle tabular representations of neural architectures.
• To improve prediction accuracy and architecture search efficiency by leveraging GBDT’s superior performance on tabular data.
• To develop a method that combines GBDT-based search space pruning with GBDT-based architecture prediction for more efficient NAS.
• To reduce the search space effectively using feature importance from GBDT and the SHAP interpretability tool.
• To demonstrate the effectiveness of GBDT in NAS through experiments on NASBench-101 and ImageNet.

Proposed method

• Employing Gradient Boosting Decision Trees (GBDT) as the accuracy predictor for neural architecture search, leveraging its ability to model tabular data representations of architectures.
• Using SHAP (SHapley Additive exPlanations) to extract feature importance from trained GBDT models to guide search space pruning.
• Gradually pruning the search space by removing low-importance architectural features identified via SHAP, resulting in a more compact and efficient search space.
• Performing NAS in two stages: first pruning the search space using GBDT as a pruner, then searching architectures using GBDT as a predictor.
• Training the GBDT predictor on existing accuracy data from the NAS benchmark to predict the performance of candidate architectures.
• Optimizing the search process by focusing on high-impact architectural components identified through GBDT's feature importance analysis.

Experimental results

Research questions

• RQ1 Can GBDT-based predictors outperform neural network-based predictors in terms of accuracy prediction for neural architecture search?
• RQ2 Does GBDT-based search space pruning improve the efficiency and effectiveness of NAS?
• RQ3 Can the combination of GBDT-based pruning and prediction lead to better final architectures on benchmark datasets?
• RQ4 How effective is SHAP in identifying important architectural features for pruning within the NAS search space?
• RQ5 What performance gains can be achieved using GBDT-based NAS on NASBench-101 and ImageNet compared to prior methods?

Key findings

• GBDT-based NAS achieved 0.18% test regret on NASBench-101, identifying the top-10 architectures within the search space.
• The method achieved a 24.2% top-1 error rate on ImageNet when using GBDT as the predictor alone.
• By combining GBDT-based search space pruning with GBDT-based architecture search, the method reduced the top-1 error rate to 23.5% on ImageNet.
• The use of SHAP to identify important architectural features enabled effective and gradual search space pruning, improving search efficiency.
• GBDT outperformed neural network-based predictors in accuracy prediction due to its superior handling of tabular architecture representations.
• The two-stage approach—pruning followed by search—proved more efficient and effective than direct search on the full space.