[Paper Review] TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search

Yawen Duan, Xin Chen+5 more|arXiv (Cornell University)|May 25, 2021

Advanced Neural Network Applications34 references4 citations

TL;DR

TransNAS-Bench-101 introduces a comprehensive benchmark for cross-task neural architecture search (NAS), evaluating 7,352 neural architectures across seven diverse vision tasks—including classification, segmentation, autoencoding, and self-supervised learning—on two search spaces: cell-level and macro-level. The key contribution is a large-scale, fully trained dataset with performance, FLOPs, and inference time, enabling robust evaluation of transferability and generalizability in NAS, with REA-transfer emerging as the top-performing transfer method, achieving 99.11th percentile performance on the macro-level search space.

ABSTRACT

Recent breakthroughs of Neural Architecture Search (NAS) extend the field's research scope towards a broader range of vision tasks and more diversified search spaces. While existing NAS methods mostly design architectures on a single task, algorithms that look beyond single-task search are surging to pursue a more efficient and universal solution across various tasks. Many of them leverage transfer learning and seek to preserve, reuse, and refine network design knowledge to achieve higher efficiency in future tasks. However, the enormous computational cost and experiment complexity of cross-task NAS are imposing barriers for valuable research in this direction. Existing NAS benchmarks all focus on one type of vision task, i.e., classification. In this work, we propose TransNAS-Bench-101, a benchmark dataset containing network performance across seven tasks, covering classification, regression, pixel-level prediction, and self-supervised tasks. This diversity provides opportunities to transfer NAS methods among tasks and allows for more complex transfer schemes to evolve. We explore two fundamentally different types of search space: cell-level search space and macro-level search space. With 7,352 backbones evaluated on seven tasks, 51,464 trained models with detailed training information are provided. With TransNAS-Bench-101, we hope to encourage the advent of exceptional NAS algorithms that raise cross-task search efficiency and generalizability to the next level. Our dataset file will be available at Mindspore, VEGA.

Motivation & Objective

• To address the lack of comprehensive benchmarks that support cross-task neural architecture search beyond image classification.
• To reduce the computational barrier for researching transferable and generalizable NAS by providing a large-scale, fully trained dataset across diverse vision tasks.
• To evaluate and compare transfer learning strategies in NAS, particularly policy and architecture transfer, across multiple tasks and search spaces.
• To identify key challenges in transferable NAS, such as negative transfer and task-shift sensitivity, and provide actionable insights for future algorithm design.
• To encourage the development of more efficient, robust, and generalizable NAS algorithms that perform well across diverse tasks and search spaces.

Proposed method

• Designing two distinct search spaces: a cell-level search space where each cell is a differentiable DAG, and a macro-level search space that allows flexible network depth and block operation configuration.
• Training and evaluating 7,352 neural architecture backbones on seven diverse vision tasks: object classification, scene classification, semantic segmentation, autoencoding, room layout prediction, surface normal estimation, and jigsaw puzzle solving.
• Providing detailed diagnostic information for each model, including accuracy, FLOPs, inference time, and training logs, to support reproducible and in-depth analysis.
• Implementing and evaluating four transfer schemes: direct architecture transfer, direct policy transfer (PPO), CATCH (a policy transfer method with encoder and evaluator components), and evolutionary algorithm-based transfer (REA) with pre-trained population initialization.
• Using a consistent experimental protocol across all tasks and search spaces to ensure fair comparison and reliable benchmarking of transfer learning performance.
• Leveraging a total of 193,760 GPU hours (equivalent to ~22.12 years on a single V100) to pre-train and evaluate all models, significantly reducing the burden for future research.

Figure 1: Our cell-level and macro-level search space in TransNAS-Bench-101. We design a cell-level search space that treats each cell as a DAG and a macro-level search space that allows flexible macro skeleton network design.

Experimental results

Research questions

• RQ1 How does the performance of NAS algorithms vary when transferring knowledge across diverse vision tasks, including classification, regression, and self-supervised learning?
• RQ2 To what extent do different transfer strategies—such as direct architecture transfer, policy transfer, and evolutionary-based initialization—improve generalization and efficiency in cross-task NAS?
• RQ3 How do cell-level and macro-level search spaces compare in terms of transferability and performance robustness across different tasks?
• RQ4 What are the key challenges in maintaining consistent performance when transferring NAS policies or architectures across tasks with significantly different characteristics?
• RQ5 Can added components in transfer methods (e.g., task encoders and evaluators in CATCH) effectively mitigate negative transfer and improve robustness?

Key findings

• Direct transfer of architectures from a source task to a target task performs significantly worse than random search, indicating high risk of negative transfer.
• Direct policy transfer (e.g., PPO) outperforms direct architecture transfer but still suffers from negative transfer, especially when the source and target tasks differ substantially.
• CATCH, which incorporates a task encoder and candidate evaluator, improves transfer performance on specific tasks such as object classification in the macro-level search space, though it struggles on others like room layout in the cell-level space.
• Evolutionary algorithm-based transfer (REA) with pre-trained population initialization achieves the highest average performance across all tasks, reaching the 99.02nd and 99.11th percentiles on the cell-level and macro-level search spaces, respectively.
• The benchmark reveals that network macro-structure design has a larger impact on performance than cell-level design for certain tasks, suggesting that NAS research should extend beyond cell-based search spaces.
• There remains significant room for improvement in NAS generalizability, as no single method consistently outperforms others across all tasks and search spaces, highlighting the need for more adaptive and robust transfer mechanisms.

Figure 2: Vision tasks considered in our benchmarks. We carefully select those 7 tasks above to ensure both diversity and similarity across tasks from Taskonomy [ 35 ] .