[Paper Review] Learning to reinforcement learn for Neural Architecture Search

Jorge Gomez-Robles, Joaquin Vanschoren|arXiv (Cornell University)|Nov 9, 2019

Adversarial Robustness in Machine Learning16 references14 citations

TL;DR

This paper proposes a deep meta-reinforcement learning (meta-RL) framework for Neural Architecture Search (NAS) that learns a transferable policy across multiple image classification datasets. By adapting previously learned strategies instead of retraining from scratch, the agent achieves better performance under resource constraints and generalizes effectively to unseen datasets, outperforming standard RL in architecture quality while demonstrating adaptive behavior during training.

ABSTRACT

Reinforcement learning (RL) is a goal-oriented learning solution that has proven to be successful for Neural Architecture Search (NAS) on the CIFAR and ImageNet datasets. However, a limitation of this approach is its high computational cost, making it unfeasible to replay it on other datasets. Through meta-learning, we could bring this cost down by adapting previously learned policies instead of learning them from scratch. In this work, we propose a deep meta-RL algorithm that learns an adaptive policy over a set of environments, making it possible to transfer it to previously unseen tasks. The algorithm was applied to various proof-of-concept environments in the past, but we adapt it to the NAS problem. We empirically investigate the agent's behavior during training when challenged to design chain-structured neural architectures for three datasets with increasing levels of hardness, to later fix the policy and evaluate it on two unseen datasets of different difficulty. Our results show that, under resource constraints, the agent effectively adapts its strategy during training to design better architectures than the ones designed by a standard RL algorithm, and can design good architectures during the evaluation on previously unseen environments. We also provide guidelines on the applicability of our framework in a more complex NAS setting by studying the progress of the agent when challenged to design multi-branch architectures.

Motivation & Objective

• To address the high computational cost of training reinforcement learning (RL) agents from scratch for each new dataset in NAS.
• To enable transfer learning in NAS by developing a meta-RL agent that learns a generalizable policy across diverse datasets.
• To evaluate whether the meta-RL agent can adapt its strategy during training and generalize to previously unseen datasets without further training.
• To explore the scalability of the framework to more complex architectures, such as multi-branch neural networks.
• To provide practical guidelines for applying meta-RL in real-world NAS settings under resource constraints.

Proposed method

• The method employs a deep meta-RL algorithm that learns a stochastic policy over a set of environments representing different image classification datasets with varying difficulty levels.
• The agent uses a recurrent neural network (LSTM) to model temporal dependencies in the architecture search process, enabling it to maintain context across search steps.
• The policy is trained using an adapted Advantage Actor-Critic (A2C) algorithm with meta-learning objectives that optimize for performance across multiple tasks.
• The framework uses a meta-dataset with three progressively harder datasets (CIFAR-10, CIFAR-100, ImageNet-100) to simulate diverse search environments.
• During evaluation, the fixed policy is applied to two new, previously unseen datasets to test zero-shot generalization capability.
• The action space is defined over chain-structured neural architectures, with operations selected based on a predefined search space of basic blocks.

Experimental results

Research questions

• RQ1 Can a meta-RL agent learn a transferable policy that generalizes across multiple image classification datasets in NAS?
• RQ2 Does the meta-RL agent adapt its strategy during training based on the difficulty of the dataset, even under resource constraints?
• RQ3 How does the performance of the meta-RL agent compare to standard RL when both are evaluated on unseen datasets?
• RQ4 Can the meta-RL framework be extended to more complex architectures, such as multi-branch neural networks?
• RQ5 What hyperparameter configurations and architectural modifications improve the agent’s exploration and adaptation in complex search spaces?

Key findings

• Under resource constraints, the meta-RL agent outperformed standard RL in designing better-performing neural architectures, demonstrating effective adaptation during training.
• The meta-RL agent exhibited adaptive behavior during training, modifying its strategy based on the difficulty of the dataset, which contributed to improved performance.
• When evaluated on two previously unseen datasets, the fixed policy generated architectures that achieved higher and more consistent accuracy than those from human-designed baselines.
• In the multi-branch setting, the agent failed to adequately explore branching structures and defaulted to chain-structured networks, indicating limitations in exploration.
• The learning progress of the meta-RL agent was slow, with a long initial exploration phase, suggesting that hyperparameter tuning—particularly for j, η, and α—could improve training efficiency.
• The observed low accuracy in network performance across datasets was attributed to computational relaxations in dataset size and reward estimation, indicating that cost-reduction strategies may impact final model quality.