[Paper Review] A System for Massively Parallel Hyperparameter Tuning

Liam Li, Kevin Jamieson+5 more|arXiv (Cornell University)|Oct 13, 2018

Machine Learning and Data Classification35 references67 citations

TL;DR

The paper introduces ASHA, an asynchronous, scalable hyperparameter optimization algorithm that aggressively early-stops poorly performing configurations to enable massive parallelism, and discusses productionizing it in Determined AI.

ABSTRACT

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.

Motivation & Objective

• Motivate the need for production-quality, massively parallel hyperparameter optimization systems.
• Propose a simple, robust algorithm (ASHA) that exploits parallelism and aggressive early stopping.
• Empirically compare ASHA to state-of-the-art methods in sequential and parallel settings.
• Demonstrate production-oriented design decisions for integrating ASHA in a ML platform.

Proposed method

• Describe Successive Halving (SHA) and its limitations in parallel, large-scale regimes.
• Introduce Asynchronous SHA (ASHA) to promote configurations without waiting for full rung completion.
• Provide a formal algorithm for ASHA including promotion and job scheduling rules.
• Compare SHA/ASHA to PBT, Vizier, BOHB, and other methods in sequential and parallel experiments.
• Show applicability of ASHA to NAS-style and large-scale language model tuning.

Experimental results

Research questions

• RQ1 How can SHA be adapted to the parallel, large-scale regime to minimize latency and maximize throughput?
• RQ2 Does an asynchronous promotion strategy (ASHA) outperform synchronous SHA and other baselines in distributed settings?
• RQ3 How does ASHA perform across CNN NAS tasks, RNN NAS tasks, and large-scale language modeling tasks?
• RQ4 What production-design decisions are needed to deploy ASHA in a real ML platform?
• RQ5 What are the trade-offs between aggressive early-stopping and robustness to mispromotions in practice?

Key findings

• ASHA yields faster configurations discovery than synchronous SHA and outperforms PBT, BOHB, and Vizier on several benchmarks.
• ASHA scales linearly with the number of workers and achieves substantial speedups in distributed settings.
• ASHA outperforms alternatives in NAS benchmarks for CNN and RNN architectures and improves perplexity and accuracy metrics.
• In large-scale language model tuning with up to 500 workers, ASHA finds good configurations in about time(R), and is roughly 3x faster than Vizier for target perplexity.
• ASHA’s production-oriented design improves usability, autoscaling, scheduling, and reproducibility in a real ML platform.