[論文レビュー] DeepArchitect: Automatically Designing and Training Deep Architectures

Renato Negrinho, Geoffrey J. Gordon|arXiv (Cornell University)|Apr 28, 2017

Machine Learning and Data Classification参考文献 27被引用数 143

ひとこと要約

tldr: A modular framework to automatically design and train deep architectures by representing architecture and training hyperparameters as a tree-structured search space, and using search strategies like random search, Monte Carlo Tree Search, and SMBO.

ABSTRACT

In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture is done manually by the human expert through a slow trial and error process guided mainly by intuition. In this paper we describe a framework for automatically designing and training deep models. We propose an extensible and modular language that allows the human expert to compactly represent complex search spaces over architectures and their hyperparameters. The resulting search spaces are tree-structured and therefore easy to traverse. Models can be automatically compiled to computational graphs once values for all hyperparameters have been chosen. We can leverage the structure of the search space to introduce different model search algorithms, such as random search, Monte Carlo tree search (MCTS), and sequential model-based optimization (SMBO). We present experiments comparing the different algorithms on CIFAR-10 and show that MCTS and SMBO outperform random search. In addition, these experiments show that our framework can be used effectively for model discovery, as it is possible to describe expressive search spaces and discover competitive models without much effort from the human expert. Code for our framework and experiments has been made publicly available.

研究の動機と目的

• Provide a programmable, compositional language to specify expressive search spaces over deep architectures and training hyperparameters.
• Enable automatic compilation of specified models to computational graphs.
• Evaluate and compare model search algorithms (random search, MCTS, SMBO) within the framework.
• Demonstrate that structured search strategies outperform random search in architecture discovery.

提案手法

• Define a modular computational module language where basic and composite modules compose to form a search space for architectures and training hyperparameters.
• Use tree-structured search spaces to traverse all fully specified models by sequentially assigning hyperparameter values.
• Provide a compilation mechanism that maps a fully specified module to a computational graph automatically.
• Implement and compare model search algorithms (Random Search, Monte Carlo Tree Search, MCTS with tree restructuring, SMBO) within the framework.
• Evaluate models on CIFAR-10 with data augmentation and fixed training budgets to compare search strategies.
• Employ surrogate modeling (ridge regression on n-gram features of module sequences) for SMBO to guide searches.

実験結果

リサーチクエスチョン

• RQ1 Can a modular, compositional language effectively express rich search spaces over architectures and training hyperparameters?
• RQ2 Do structured search algorithms (MCTS, SMBO) outperform random search in discovering high-performing architectures within the DeepArchitect framework?
• RQ3 How does tree restructuring (bisecting hyperparameter search) affect the efficiency and effectiveness of MCTS in architecture search?
• RQ4 What are the practical gains when jointly optimizing architecture and training hyperparameters for a standard dataset like CIFAR-10?

主な発見

• All search strategies can reach around 89% validation accuracy on CIFAR-10 after 64 evaluations in the tested setup.
• MCTS and SMBO outperform random search as evaluation count grows, with MCTS (bisected) outperforming random search around 32 evaluations and SMBO around 16 evaluations.
• MCTS without tree restructuring does not outperform random search in this setting, highlighting the benefit of bisecting hyperparameters with large value sets.
• Tree restructuring enables sharing information across related hyperparameter values, improving search efficiency for high-variance architectural choices.
• The framework can explore expressive search spaces and discover competitive models with limited human effort.