[Paper Review] AutoML from Service Provider's Perspective: Multi-device, Multi-tenant Model Selection with GP-EI

Chen Yu, Bojan Karlaš+3 more|arXiv (Cornell University)|Mar 17, 2018

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a multi-device, multi-tenant GP-EI algorithm for AutoML service providers to optimize global user satisfaction by intelligently allocating limited computation devices across multiple users. It achieves near-linear speedup and significantly reduces regret compared to random or round-robin scheduling, with a theoretical regret bound of $O((\text{MIU}(T,K)+M)\frac{N^2}{M})$.

ABSTRACT

AutoML has become a popular service that is provided by most leading cloud service providers today. In this paper, we focus on the AutoML problem from the \emph{service provider's perspective}, motivated by the following practical consideration: When an AutoML service needs to serve {\em multiple users} with {\em multiple devices} at the same time, how can we allocate these devices to users in an efficient way? We focus on GP-EI, one of the most popular algorithms for automatic model selection and hyperparameter tuning, used by systems such as Google Vizer. The technical contribution of this paper is the first multi-device, multi-tenant algorithm for GP-EI that is aware of \emph{multiple} computation devices and multiple users sharing the same set of computation devices. Theoretically, given $N$ users and $M$ devices, we obtain a regret bound of $O(( ext{\bf {MIU}}(T,K) + M)\frac{N^2}{M})$, where $ ext{\bf {MIU}}(T,K)$ refers to the maximal incremental uncertainty up to time $T$ for the covariance matrix $K$. Empirically, we evaluate our algorithm on two applications of automatic model selection, and show that our algorithm significantly outperforms the strategy of serving users independently. Moreover, when multiple computation devices are available, we achieve near-linear speedup when the number of users is much larger than the number of devices.

Motivation & Objective

• Address the challenge of efficient resource allocation in AutoML services where multiple users share a limited number of computation devices.
• Develop a cost-sensitive, scalable algorithm that maximizes the global 'happiness' of all users by minimizing cumulative regret across all users.
• Provide a theoretical regret bound for multi-device, multi-tenant GP-EI in the context of shared resource allocation.
• Improve upon naive scheduling strategies like random or round-robin assignment in real-world AutoML workloads.
• Enable near-linear speedup in convergence time when multiple devices are available, especially when the number of users exceeds the number of devices.

Proposed method

• The proposed MM-GP-EI algorithm uses Maximum Incremental Uncertainty (MIU) to model dependence between users' Gaussian process priors and prioritize users based on expected improvement.
• Each user maintains their own GP-EI instance with a shared prior over hyperparameters, and the system selects the next user to serve based on a weighted acquisition function that accounts for MIU and device availability.
• An upper confidence bound-style acquisition function is used to balance exploration and exploitation across users, with dynamic reweighting based on uncertainty and expected gain.
• The algorithm dynamically assigns devices to users based on the highest expected improvement per unit time, ensuring efficient use of limited hardware resources.
• An efficient implementation uses a shared covariance matrix and incremental updates to MIU to reduce computational overhead during scheduling decisions.
• Regret is minimized by prioritizing users with the highest potential for performance gain, especially in early stages when uncertainty is high.

Experimental results

Research questions

• RQ1 How can we design a multi-device, multi-tenant GP-EI algorithm that optimizes global user satisfaction in shared AutoML environments?
• RQ2 What is the theoretical regret bound of such an algorithm under shared device constraints?
• RQ3 How does the proposed algorithm compare to random and round-robin scheduling in terms of convergence speed and regret reduction?
• RQ4 Does the algorithm achieve near-linear speedup when scaling up the number of devices?
• RQ5 Under what conditions does the algorithm show the most significant performance gains compared to baseline strategies?

Key findings

• Multi-tenant GP-EI outperforms random and round-robin scheduling by up to 5× faster convergence to the same level of instantaneous regret on the Azure dataset.
• On the Azure dataset, the proposed method achieved a 5× speedup in reaching a target instantaneous regret level compared to round-robin scheduling.
• Speedup was most pronounced when the number of users exceeded the number of devices, with near-linear convergence improvement observed as device count increased.
• Experiments on a synthetic dataset with 50 users and 50 models showed that convergence time dropped at a near-linear rate with increasing device count.
• The theoretical regret bound of $O((\text{MIU}(T,K)+M)\frac{N^2}{M})$ converges to the optimal rate and supports the observed speedup behavior.
• Performance gains were smaller on the DeepLearning dataset due to lower initial accuracy variance (0.04 vs. 0.12 on Azure), indicating less room for improvement after initial trials.