[Paper Review] pySOT and POAP: An event-driven asynchronous framework for surrogate optimization

David Eriksson, David Bindel+1 more|arXiv (Cornell University)|Jul 30, 2019

Advanced Multi-Objective Optimization AlgorithmsComputer Science38 references67 citations

TL;DR

POAP provides an event-driven asynchronous framework for building and coordinating surrogate-optimization strategies, while pySOT offers a suite of surrogate optimization methods built on POAP, enabling synchronous or asynchronous global optimization of expensive black-box functions.

ABSTRACT

This paper describes Plumbing for Optimization with Asynchronous Parallelism (POAP) and the Python Surrogate Optimization Toolbox (pySOT). POAP is an event-driven framework for building and combining asynchronous optimization strategies, designed for global optimization of expensive functions where concurrent function evaluations are useful. POAP consists of three components: a worker pool capable of function evaluations, strategies to propose evaluations or other actions, and a controller that mediates the interaction between the workers and strategies. pySOT is a collection of synchronous and asynchronous surrogate optimization strategies, implemented in the POAP framework. We support the stochastic RBF method by Regis and Shoemaker along with various extensions of this method, and a general surrogate optimization strategy that covers most Bayesian optimization methods. We have implemented many different surrogate models, experimental designs, acquisition functions, and a large set of test problems. We make an extensive comparison between synchronous and asynchronous parallelism and find that the advantage of asynchronous computation increases as the variance of the evaluation time or number of processors increases. We observe a close to linear speed-up with 4, 8, and 16 processors in both the synchronous and asynchronous setting.

Motivation & Objective

• Motivate and address global optimization of expensive black-box functions with varying evaluation times.
• Propose an asynchronous, fault-resilient framework (POAP) for coordinating workers and strategies.
• Provide a Python surrogate-optimization toolbox (pySOT) built on POAP with multiple models and designs.
• Evaluate synchronous vs asynchronous parallelism and demonstrate benefits under time variance and larger processor counts.

Proposed method

• Introduce POAP as an event-driven framework with a worker pool, strategies, and a controller to manage asynchronous evaluations.
• Implement pySOT as a collection of surrogate optimization strategies (RBFs, GP, SVR, MARS, polynomial regression) within POAP.
• Describe surrogate-model options (RBF with regularization, Gaussian processes, and others) and experimental designs (LHD, SLHD).
• Detail auxiliary problems (selection of candidate points via LMS-RBF, DYCORS, PI/EI/LCB acquisition in BO) and dynamic sampling radius updates.
• Explain asynchronous algorithm (Algorithm 3) and how it handles evaluation events, worker availability, and restarts (Algorithm 4).
• Discuss implementation components (controllers, strategies, workers) and interfaces for asynchronous optimization, including MPI and simulated-time testing.

Experimental results

Research questions

• RQ1 How does asynchronous (vs synchronous) parallelism impact performance in surrogate-based global optimization when evaluation times vary and more processors are used?
• RQ2 Can POAP provide fault resilience and dynamic worker reallocation to improve robustness of asynchronous surrogate optimization?
• RQ3 How do different surrogate models, experimental designs, and acquisition functions perform within an event-driven asynchronous framework?
• RQ4 What are practical guidelines for restarting, updating sampling radii, and combining strategies in asynchronous settings?

Key findings

• Asynchrony offers near-linear speed-up with 4, 8, and 16 processors in both synchronous and asynchronous setups under varying evaluation times.
• Asynchronous computation reduces idle time and its advantage grows with higher evaluation variance and more processors.
• Parallelism improves exploration and can outperform serial runs in terms of number of evaluations.
• POAP provides fault resilience and dynamic worker management, handling crashes or failures gracefully.
• pySOT serves as a robust test-suite for comparing surrogate models, designs, and acquisition functions within the POAP framework.
• The combined POAP-pySOT framework has achieved substantial adoption, with pySOT downloaded over 88,000 times and POAP over 126,000 times.