[Paper Review] Integrating Multi-Armed Bandit, Active Learning, and Distributed Computing for Scalable Optimization

Foo Hui-Mean, Yuan‐chin Ivan Chang|arXiv (Cornell University)|Jan 2, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

ALMAB-DC is a modular framework that unifies active learning, multi-armed bandits, and distributed computing to enable scalable, uncertainty-aware black-box optimization with GPU acceleration.

ABSTRACT

Modern optimization problems in scientific and engineering domains often rely on expensive black-box evaluations, such as those arising in physical simulations or deep learning pipelines, where gradient information is unavailable or unreliable. In these settings, conventional optimization methods quickly become impractical due to prohibitive computational costs and poor scalability. We propose ALMAB-DC, a unified and modular framework for scalable black-box optimization that integrates active learning, multi-armed bandits, and distributed computing, with optional GPU acceleration. The framework leverages surrogate modeling and information-theoretic acquisition functions to guide informative sample selection, while bandit-based controllers dynamically allocate computational resources across candidate evaluations in a statistically principled manner. These decisions are executed asynchronously within a distributed multi-agent system, enabling high-throughput parallel evaluation. We establish theoretical regret bounds for both UCB-based and Thompson-sampling-based variants and develop a scalability analysis grounded in Amdahl's and Gustafson's laws. Empirical results across synthetic benchmarks, reinforcement learning tasks, and scientific simulation problems demonstrate that ALMAB-DC consistently outperforms state-of-the-art black-box optimizers. By design, ALMAB-DC is modular, uncertainty-aware, and extensible, making it particularly well suited for high-dimensional, resource-intensive optimization challenges.

Motivation & Objective

• Address the challenge of expensive black-box evaluations in high-dimensional settings without reliable gradients.
• Develop a unified, modular framework that combines active learning, MAB, and distributed computing for scalable optimization.
• Leverage surrogate modeling and information-theoretic acquisition to guide informative sampling under budget constraints.
• Enable asynchronous, GPU-accelerated evaluation across distributed agents to improve throughput.
• Provide theoretical regret bounds and scalability analyses to support distributed, uncertainty-aware optimization.

Proposed method

• Treat optimization as a sequential decision-making process under uncertainty guided by Bayesian surrogates.
• Use acquisition functions (e.g., entropy, expected improvement, mutual information) to select the next informative input.
• Incorporate bandit-based controllers (UCB, Thompson Sampling) to allocate computational resources across candidate evaluations.
• Distribute evaluations asynchronously across compute nodes with optional GPU acceleration for surrogate modeling and evaluation.
• Update the surrogate model and bandit statistics iteratively to refine future queries.
• Provide theoretical regret bounds and scalability analyses for distributed, asynchronous settings.

Figure 1: ALMAB-DC Framework: Integration of Active Learning, Multi-Armed Bandits, and Distributed Computing through Bayesian Surrogate Modeling

Experimental results

Research questions

• RQ1 How can active learning and bandit strategies be integrated to enable scalable, information-efficient black-box optimization in distributed environments?
• RQ2 What are the theoretical regret and scalability properties of ALMAB-DC under asynchronous feedback and communication overhead?
• RQ3 How does GPU-accelerated, distributed evaluation influence throughput and convergence in high-cost optimization tasks?
• RQ4 What is the optimal balance between number of agents and coordination costs for scalable performance?
• RQ5 Can ALMAB-DC adapt to multi-fidelity and heterogeneous computing settings while preserving uncertainty quantification?

Key findings

• ALMAB-DC achieves improved scalability and sample efficiency by unifying AL, MAB, and DC in a modular pipeline.
• The framework provides regret bounds for UCB- and Thompson Sampling-based variants in distributed, asynchronous settings.
• A scalability analysis based on Amdahl’s and Gustafson’s laws guides resource allocation and predicts speedups and limitations.
• GPU acceleration is leveraged for surrogate modeling, posterior updates, and acquisition optimization to boost throughput.
• Empirical results across synthetic benchmarks, reinforcement learning tasks, and scientific simulations show competitive performance against state-of-the-art black-box optimizers.

Figure 2: ALMAB-DC Architecture Pipeline: The framework integrates Active Learning (AL), Multi-Armed Bandits (MAB), and Distributed Computing (DC) into a modular pipeline. Decision modules (top) include the Unlabeled Data Pool, Active Learner, and Bandit Controller, which guide candidate selection a