[Paper Review] An extensive numerical benchmark study of deterministic vs. stochastic derivative-free global optimization algorithms

Linas Stripinis, Remigijus Paulavičius|arXiv (Cornell University)|Sep 13, 2022

Advanced Optimization Algorithms Research5 citations

TL;DR

This paper presents a large-scale benchmark of 64 derivative-free global optimization algorithms—64 deterministic and stochastic—on 1,197 test problems from the GKLS and DIRECTGOLib v1.2 collections. It finds that deterministic DIRECT-type algorithms outperform stochastic ones on low-dimensional and GKLS-type problems, while stochastic solvers like AVOA dominate in high-dimensional settings, especially under limited function evaluation budgets.

ABSTRACT

Research in derivative-free global optimization is under active development, and many solution techniques are available today. Therefore, the experimental comparison of previous and emerging algorithms must be kept up to date. This paper considers the solution to the bound-constrained, possibly black-box global optimization problem. It compares 64 derivative-free deterministic algorithms against classic and state-of-the-art stochastic solvers. Among deterministic ones, a particular emphasis is on DIRECT-type, where, in recent years, significant progress has been made. A set of 800 test problems generated by the well-known GKLS generator and 397 traditional test problems from DIRECTGOLib v1.2 collection are utilized in a computational study. More than 239400 solver runs were carried out, requiring more than 531 days of single CPU time to complete them. It has been found that deterministic algorithms perform excellently on GKLS-type and low-dimensional problems, while stochastic algorithms have shown to be more efficient in higher dimensions.

Motivation & Objective

• To provide a comprehensive, up-to-date comparison of state-of-the-art deterministic and stochastic derivative-free global optimization algorithms.
• To evaluate algorithm performance across diverse, challenging test problems, including both synthetic (GKLS) and traditional (DIRECTGOLib v1.2) benchmarks.
• To assess the impact of function evaluation budget and problem dimensionality on algorithm effectiveness.
• To investigate the performance of hybridized versus non-hybridized variants of deterministic solvers, particularly in DIRECT-type algorithms.
• To guide practitioners in selecting optimal solvers based on problem characteristics such as dimensionality and structure.

Proposed method

• Utilized 800 GKLS-generated test problems and 397 traditional problems from DIRECTGOLib v1.2, totaling 1,197 problems.
• Executed over 239,400 solver runs across multiple function evaluation budgets: $M_{\text{max}} = 2,500$ and $M_{\text{max}} = 500,000$.
• Evaluated 64 solvers, including 34 deterministic (especially DIRECT-type) and 30 stochastic (e.g., AVOA, SCSO, BESO, CA), using a standardized computational framework.
• Applied performance metrics based on convergence to global optimum and function evaluation efficiency, with results aggregated across all test problems.
• Used the DIRECTGOLib v1.2 repository as a publicly available, extensible test suite under MIT license, ensuring reproducibility and community contribution.
• Compared hybridized versions (e.g., OQNLP*, MCS*) against standard counterparts to assess the impact of hybridization on performance.

Experimental results

Research questions

• RQ1 How do deterministic and stochastic derivative-free optimization algorithms compare in performance across low- and high-dimensional problems?
• RQ2 Which algorithms perform best on the GKLS test set, particularly under high function evaluation budgets?
• RQ3 How does the inclusion of hybridized variants affect the performance of deterministic DIRECT-type solvers compared to their non-hybridized counterparts?
• RQ4 What is the relative efficiency of stochastic solvers like AVOA and SCSO on high-dimensional problems compared to deterministic methods?
• RQ5 To what extent do algorithm rankings depend on problem dimensionality and function evaluation budget?

Key findings

• Deterministic DIRECT-type algorithms, particularly BIRMIN* and Gb-BIRECT, outperformed all others on low-dimensional and GKLS-type problems, especially under $M_{\text{max}} = 500,000$.
• Stochastic solvers such as AVOA, SCSO, BESO, and CA were the top performers on high-dimensional problems when $M_{\text{max}} = 2,500$, indicating superior scalability.
• AVOA ranked tenth overall in the full benchmark, despite being the best in high-dimensional settings, highlighting the strong influence of GKLS problems on overall rankings.
• Hybridized deterministic solvers (e.g., OQNLP*, MCS*, MSNLP*, BIRMIN*) underperformed compared to their non-hybridized counterparts, even at $M_{\text{max}} = 500,000$, suggesting that hybridization may impair global search efficiency.
• The performance of algorithms was highly problem-dependent: no single algorithm dominated across all test problems, and rankings varied significantly by dimensionality and function type.
• The study confirms that DIRECT-type solvers are highly effective in low dimensions but lose efficiency rapidly as dimensionality increases, consistent with known theoretical limitations.