[Paper Review] Automated Machine Learning: State-of-The-Art and Open Challenges

Radwa El Shawi, Mohamed Maher+1 more|arXiv (Cornell University)|Jun 5, 2019

Machine Learning and Data Classification113 references152 citations

TL;DR

This paper surveys AutoML approaches addressing CASH, surveys meta-learning, NAS, hyper-parameter optimization, and AutoML tools, and discusses open challenges and future directions.

ABSTRACT

With the continuous and vast increase in the amount of data in our digital world, it has been acknowledged that the number of knowledgeable data scientists can not scale to address these challenges. Thus, there was a crucial need for automating the process of building good machine learning models. In the last few years, several techniques and frameworks have been introduced to tackle the challenge of automating the process of Combined Algorithm Selection and Hyper-parameter tuning (CASH) in the machine learning domain. The main aim of these techniques is to reduce the role of the human in the loop and fill the gap for non-expert machine learning users by playing the role of the domain expert. In this paper, we present a comprehensive survey for the state-of-the-art efforts in tackling the CASH problem. In addition, we highlight the research work of automating the other steps of the full complex machine learning pipeline (AutoML) from data understanding till model deployment. Furthermore, we provide comprehensive coverage for the various tools and frameworks that have been introduced in this domain. Finally, we discuss some of the research directions and open challenges that need to be addressed in order to achieve the vision and goals of the AutoML process.

Motivation & Objective

• Survey the state-of-the-art methods for Combined Algorithm Selection and Hyper-parameter tuning (CASH) in AutoML.
• Highlight automatic processing of end-to-end ML pipelines from data understanding to deployment.
• Provide coverage of tools and frameworks enabling AutoML.
• Discuss open challenges and future research directions in AutoML.

Proposed method

• Present a taxonomy and taxonomy-driven survey of CASH and AutoML approaches.
• Summarize meta-learning techniques for warm-starting AutoML searches.
• Describe Neural Architecture Search (NAS) categories and methods.
• Review automated hyper-parameter optimization techniques (black-box and multi-fidelity).
• Catalog tools and frameworks for AutoML and their design choices (centralized, distributed, cloud-based).
• Outline pre-modeling and post-modeling automation steps in the ML pipeline.

Experimental results

Research questions

• RQ1 What are the current state-of-the-art approaches to CASH and how do they balance performance with search cost?
• RQ2 How do meta-learning, NAS, and hyper-parameter optimization contribute to AutoML efficiency and effectiveness?
• RQ3 What frameworks and tools exist for AutoML, and what are their comparative strengths and limitations?
• RQ4 What are the remaining open challenges in automating the full ML pipeline from data understanding to deployment?

Key findings

• Meta-learning and prior-task information are used to warm-start AutoML searches and reduce search time.
• NAS techniques fall into five categories: random search, reinforcement learning, gradient-based, evolutionary, and Bayesian optimization, with varied success across tasks.
• Hyper-parameter optimization is categorized into black-box and multi-fidelity approaches, including Bayesian optimization, TPE, SMAC, SA, GA, and learning-curve-based termination.
• Multiple AutoML tools/frameworks exist (Auto-Weka, Auto-Sklearn, TPOT, ML-Plan, Hyperopt-Sklearn, Recipe, Auto-MEKA_GGP, SmartML) with different design choices like meta-learning, ensembles, grammar-based search, and hierarchical pipelines.
• The paper provides a repository of AutoML resources and discusses end-to-end automation from data understanding to deployment.
• Open challenges include scalability, efficiency, and the integration of diverse pipeline components across different domains.