[Paper Review] A Review of Meta-Reinforcement Learning for Deep Neural Networks Architecture Search

Yesmina Jaafra, Jean Luc Laurent+2 more|arXiv (Cornell University)|Dec 17, 2018

Advanced Neural Network Applications46 references11 citations

TL;DR

This paper reviews meta-reinforcement learning (meta-RL) approaches for automating convolutional neural network (CNN) architecture search, focusing on how meta-RL enables efficient, adaptive controller policies that learn to search across vast architecture spaces. The key contribution is a comprehensive synthesis of current methods, highlighting reinforcement learning as the dominant strategy, modular search spaces, and performance prediction techniques that reduce computational cost while maintaining high accuracy in image classification tasks.

ABSTRACT

Deep Neural networks are efficient and flexible models that perform well for a variety of tasks such as image, speech recognition and natural language understanding. In particular, convolutional neural networks (CNN) generate a keen interest among researchers in computer vision and more specifically in classification tasks. CNN architecture and related hyperparameters are generally correlated to the nature of the processed task as the network extracts complex and relevant characteristics allowing the optimal convergence. Designing such architectures requires significant human expertise, substantial computation time and doesn't always lead to the optimal network. Model configuration topic has been extensively studied in machine learning without leading to a standard automatic method. This survey focuses on reviewing and discussing the current progress in automating CNN architecture search.

Motivation & Objective

• To survey and analyze recent advances in meta-reinforcement learning for automated deep neural network architecture search.
• To identify key methodological trends in search optimization, architecture design (plain vs. modular), and search acceleration techniques.
• To evaluate the role of performance prediction, proxy metrics, and network transformation in reducing computational cost during neural architecture search.
• To highlight the limitations of current approaches, particularly the risk of bias from early stopping and proxy metrics.
• To identify future research directions, including multi-task learning, weight sharing, and more efficient search algorithms for broader accessibility.

Proposed method

• Employs meta-reinforcement learning to train a controller policy that generates and evaluates CNN architectures based on performance feedback.
• Uses a parameter-server framework with distributed training across multiple GPUs and machines to parallelize evaluation of candidate architectures.
• Applies surrogate models and learning curve extrapolation to predict validation accuracy early, avoiding full training of poor architectures.
• Leverages network morphism and transfer learning to initialize new architectures with pre-trained weights, accelerating convergence.
• Implements proxy metrics such as mini-batch training and down-sampled images to estimate performance rapidly during search.
• Introduces modular (multi-branch) architecture design to constrain the search space to reusable blocks, improving efficiency and generalization.

Experimental results

Research questions

• RQ1 How can meta-reinforcement learning effectively automate the search for optimal CNN architectures across diverse image classification tasks?
• RQ2 What are the most effective search acceleration techniques that reduce computational cost without degrading final model performance?
• RQ3 To what extent do proxy metrics and early stopping introduce bias into the architecture search process?
• RQ4 How do modular design approaches (e.g., cell-based search) improve search efficiency and generalization compared to plain architecture search?
• RQ5 What future directions—such as multi-task learning or weight sharing—can make automated architecture search more accessible and scalable?

Key findings

• Reinforcement learning has emerged as the dominant strategy for training architecture search controllers, demonstrating strong empirical performance in image classification.
• Modular architecture search, particularly using reusable blocks or cells, significantly reduces the search space and improves generalization across tasks.
• Performance prediction via surrogate models and early stopping (e.g., using mini-batches or low-resolution data) reduces training time by up to 90% without major accuracy loss.
• Network morphism and transfer learning techniques enable faster convergence by reusing pre-trained weights, reducing the need for random initialization.
• Despite acceleration techniques, current meta-RL-based approaches remain computationally expensive, limiting access for small research groups.
• Future methods such as multi-task learning and weight sharing are essential to reduce computational barriers and improve scalability of automated architecture search.