[Paper Review] Multi-device, Multi-tenant Model Selection with GP-EI.

Yu Chen, Bojan Karlaš+3 more|arXiv (Cornell University)|Mar 17, 2018

Advanced Bandit Algorithms Research16 references3 citations

TL;DR

This paper proposes a multi-device, multi-tenant extension of GP-EI for Bayesian optimization in AutoML, enabling efficient resource allocation across multiple users and devices. It achieves a regret bound of $O((\text{MIU}(T,K) + M)\frac{N^2}{M})$ and demonstrates near-linear speedup when the number of users far exceeds the number of devices, significantly outperforming independent user scheduling.

ABSTRACT

Bayesian optimization is the core technique behind the emergence of AutoML, which holds the promise of automatically searching for models and hyperparameters to make machine learning techniques more accessible. As such services are moving towards the cloud, we ask -- {\em When multiple AutoML users share the same computational infrastructure, how should we allocate resources to maximize the global happiness of all users?} We focus on GP-EI, one of the most popular algorithms for automatic model selection and hyperparameter tuning, and develop a novel multi-device, multi-tenant extension that is aware of \emph{multiple} computation devices and multiple users sharing the same set of computation devices. Theoretically, given $N$ users and $M$ devices, we obtain a regret bound of $O(( ext{\bf {MIU}}(T,K) + M)\frac{N^2}{M})$, where $ ext{\bf {MIU}}(T,K)$ refers to the maximal incremental uncertainty up to time $T$ for the covariance matrix $K$. Empirically, we evaluate our algorithm on two applications of automatic model selection, and show that our algorithm significantly outperforms the strategy of serving users independently. Moreover, when multiple computation devices are available, we achieve near-linear speedup when the number of users is much larger than the number of devices.

Motivation & Objective

• To address the challenge of efficient resource allocation in shared cloud-based AutoML systems where multiple users and devices coexist.
• To extend the GP-EI algorithm to handle multi-tenant and multi-device environments while maintaining theoretical performance guarantees.
• To maximize global user satisfaction by optimizing the allocation of computational resources across users and devices.
• To achieve scalable and efficient hyperparameter tuning in distributed, shared computational infrastructures.

Proposed method

• The method extends GP-EI to support multiple users and devices by modeling inter-user and inter-device dependencies in the optimization process.
• It introduces a regret analysis framework that accounts for maximal incremental uncertainty (MIU) in the covariance matrix over time.
• The algorithm dynamically allocates computation time across devices based on user demand and uncertainty reduction potential.
• It uses a theoretical regret bound that scales with $O((\text{MIU}(T,K) + M)\frac{N^2}{M})$, where $N$ is the number of users and $M$ the number of devices.
• The approach enables parallel execution across devices while maintaining coherence in the Bayesian optimization process across tenants.

Experimental results

Research questions

• RQ1 How can Bayesian optimization be extended to support multiple users and devices in a shared infrastructure?
• RQ2 What theoretical regret bounds can be achieved in a multi-tenant, multi-device setting for GP-EI?
• RQ3 How does the proposed method scale with increasing numbers of users relative to available devices?
• RQ4 Can the algorithm achieve near-linear speedup when the number of users is much larger than the number of devices?

Key findings

• The proposed algorithm achieves a regret bound of $O((\text{MIU}(T,K) + M)\frac{N^2}{M})$, providing theoretical guarantees for multi-tenant, multi-device optimization.
• Empirical evaluation shows significant performance gains over independent user scheduling strategies in two automatic model selection applications.
• When multiple devices are available and the number of users is much larger than the number of devices, the method achieves near-linear speedup.
• The algorithm effectively balances exploration and exploitation across users while maintaining computational efficiency in shared environments.