[Paper Review] A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms

Yanan Sun, Xian Sun+2 more|arXiv (Cornell University)|Aug 30, 2020

Machine Learning and Data Classification46 references4 citations

TL;DR

This paper proposes a novel training protocol for performance predictors in evolutionary neural architecture search (ENAS) that improves prediction accuracy by using a pairwise ranking indicator as the target, logistic regression for modeling, and difference-based training instances to avoid data imbalance. Experiments show a significant 16.97% average improvement on CIFAR10-CNN and 7.49% on CIFAR100-CNN over traditional protocols.

ABSTRACT

Evolutionary Neural Architecture Search (ENAS) can automatically design the architectures of Deep Neural Networks (DNNs) using evolutionary computation algorithms. However, most ENAS algorithms require intensive computational resource, which is not necessarily available to the users interested. Performance predictors are a type of regression models which can assist to accomplish the search, while without exerting much computational resource. Despite various performance predictors have been designed, they employ the same training protocol to build the regression models: 1) sampling a set of DNNs with performance as the training dataset, 2) training the model with the mean square error criterion, and 3) predicting the performance of DNNs newly generated during the ENAS. In this paper, we point out that the three steps constituting the training protocol are not well though-out through intuitive and illustrative examples. Furthermore, we propose a new training protocol to address these issues, consisting of designing a pairwise ranking indicator to construct the training target, proposing to use the logistic regression to fit the training samples, and developing a differential method to building the training instances. To verify the effectiveness of the proposed training protocol, four widely used regression models in the field of machine learning have been chosen to perform the comparisons on two benchmark datasets. The experimental results of all the comparisons demonstrate that the proposed training protocol can significantly improve the performance prediction accuracy against the traditional training protocols.

Motivation & Objective

• Address the inefficiency and inaccuracy of traditional training protocols for performance predictors in evolutionary neural architecture search (ENAS).
• Identify flaws in the conventional approach—using mean square error with raw performance values and independent samples—as suboptimal for ranking-based prediction tasks.
• Develop a new training protocol that better aligns with real-world ENAS use cases where relative performance ranking is more critical than absolute value prediction.
• Improve the generalization and robustness of performance predictors by addressing data imbalance and modeling complexity in regression tasks.

Proposed method

• Propose a pairwise ranking indicator as the training target, which captures relative performance between DNN architectures instead of absolute values.
• Replace traditional linear regression with logistic regression to model the probability that one architecture outperforms another, improving ranking fidelity.
• Construct training instances as the difference between pairs of DNN architectures to naturally balance positive and negative samples and reduce data imbalance.
• Integrate the three components into a unified training protocol that enhances model generalization and prediction accuracy for performance predictors.
• Apply the protocol across four standard regression models (SVM, GBDT, DTree, RForest) on two benchmark datasets to validate effectiveness.
• Use ablation studies to isolate and evaluate the contribution of each component, particularly the interdependence of the first two components.

Experimental results

Research questions

• RQ1 How does the use of a pairwise ranking indicator as a training target affect the performance prediction accuracy of ENAS predictors compared to traditional mean square error-based regression?
• RQ2 To what extent does replacing linear regression with logistic regression improve the ranking capability of performance predictors in ENAS?
• RQ3 Can constructing training instances as differences between DNN architecture pairs mitigate data imbalance and enhance predictor generalization?
• RQ4 Which components of the proposed training protocol contribute most significantly to performance improvement, and how do they interact?

Key findings

• The proposed training protocol significantly improves prediction accuracy, achieving a 16.97% average improvement on the CIFAR10-CNN benchmark compared to the traditional protocol.
• On the CIFAR100-CNN benchmark, the proposed protocol yields a 7.49% average improvement over the traditional protocol, demonstrating consistent gains across datasets.
• When the first two components (pairwise ranking and logistic regression) are disabled, performance drops drastically—by 16.97% on CIFAR10-CNN and 7.49% on CIFAR100-CNN—indicating their critical role.
• Disabling only the third component (difference-based instances) results in a much smaller drop of 2.75% on CIFAR10-CNN and 1.17% on CIFAR100-CNN, showing its lesser but still beneficial contribution.
• The ablation study confirms that the first two components are interdependent and cannot function effectively in isolation, highlighting their synergistic impact.
• The proposed protocol consistently outperforms the traditional protocol across all four regression models (SVM, GBDT, DTree, RForest) on both benchmark datasets.