[Paper Review] GENDIRECT: a GENeralized DIRECT-type algorithmic framework for derivative-free global optimization

Linas Stripinis, Remigijus Paulavičius|arXiv (Cornell University)|Sep 2, 2023

Advanced Optimization Algorithms ResearchMathematics3 citations

TL;DR

This paper introduces GENDIRECT, a generalized algorithmic framework that unifies and extends DIRECT-type derivative-free global optimization methods by enabling flexible composition of algorithmic components. By reconfiguring five existing algorithms using GENDIRECT, the study demonstrates significant performance improvements, with enhanced efficiency and robustness across diverse benchmark problems, achieving statistically significant gains in over 80% of test cases.

ABSTRACT

Over the past three decades, numerous articles have been published discussing the renowned DIRECT algorithm (DIvididing RECTangles). These articles present innovative ideas to enhance its performance and adapt it to various types of optimization problems. A comprehensive collection of deterministic, derivative-free algorithmic implementations based on the DIRECT framework has recently been introduced as part of the DIRECTGO project. DIRECTGO empowers users to conveniently employ diverse DIRECT-type algorithms, enabling efficient solutions to practical optimization problems. Despite their variations, DIRECT-type algorithms share a common algorithmic structure and typically differ only at certain steps. Therefore, we propose GENDIRECT -- GENeralized DIRECT-type framework that encompasses and unifies DIRECT-type algorithms into a single, generalized framework within this paper. GENDIRECT offers a practical alternative to the creation of yet another ``new'' DIRECT-type algorithm that closely resembles existing ones. Instead, GENDIRECT allows the efficient generation of known or novel DIRECT-type optimization algorithms by assembling different algorithmic components. This approach provides considerably more flexibility compared to both the DIRECTGO toolbox and individual DIRECT-type algorithms. A few hundred thousand DIRECT-type algorithms can be combined using GENDIRECT, facilitating users' easy customization and the addition of new algorithmic components. By modifying specific components of five highly promising DIRECT-type algorithms found in the existing literature using GENDIRECT, the significant potential of GENDIRECT has been demonstrated. The resulting newly developed improved approaches exhibit greater efficiency and enhanced robustness in dealing with problems of varying complexity.

Motivation & Objective

• To address the proliferation of similar yet distinct DIRECT-type algorithms by unifying their design under a single extensible framework.
• To reduce the need for developing new algorithms from scratch by enabling rapid, modular construction of existing or novel algorithms.
• To improve the efficiency and robustness of derivative-free global optimization for black-box problems with expensive function evaluations.
• To facilitate systematic exploration of algorithmic component combinations for optimal performance.

Proposed method

• GENDIRECT formalizes DIRECT-type algorithms as a composition of interchangeable algorithmic components, such as partitioning rules, sampling strategies, and selection criteria.
• The framework supports the creation of approximately 500,000 unique algorithmic configurations through combinatorial assembly of components.
• It generalizes the core structure of DIRECT by abstracting key steps into modular functions, enabling dynamic reconfiguration.
• The framework is implemented in MATLAB and integrated with DIRECTGOLib v2.0 for benchmarking.
• Algorithmic improvements are generated by modifying specific components (e.g., partitioning, sampling) in five high-performing existing algorithms.
• Performance is evaluated using statistical tests (Wilcoxon signed-rank) and Friedman’s mean rank analysis across 20 test problems and four evaluation budgets.

Experimental results

Research questions

• RQ1 Can a generalized framework be designed to unify and extend the diversity of existing DIRECT-type algorithms?
• RQ2 To what extent can performance gains be achieved by reconfiguring algorithmic components rather than designing new algorithms from scratch?
• RQ3 Which combinations of algorithmic components yield the most robust and efficient performance across different problem classes and evaluation budgets?
• RQ4 How do the improved algorithms compare statistically to their original counterparts across varying levels of function evaluation budgets?

Key findings

• The improved 1-DTC-GL algorithm achieved a mean rank of 4.6128 at Mmax = n×104, significantly outperforming the original version (mean rank 5.3375), with p-value = 2.4241×10−8 at Mmax = n×103.
• The improved HALRECT-IA version achieved a mean rank of 5.5229 at Mmax = n×104, significantly better than the original (mean rank 7.1372), with p-value = 5.3640×10−10.
• The improved MrDIRECT algorithm achieved a mean rank of 5.2500 at Mmax = n×104, significantly outperforming the original (mean rank 6.8099), with p-value = 4.5374×10−40.
• The improved BIRMIN algorithm showed consistent statistical superiority across all budgets, with p-values below 1.5552×10−15 at Mmax = n×105.
• The improved DIRMIN algorithm achieved a mean rank of 5.2886 at Mmax = n×104, significantly better than the original (mean rank 5.3084), with p-value = 8.2874×10−3.
• Overall, 80% of the improved algorithms showed statistically significant performance gains, particularly at higher evaluation budgets (Mmax ≥ n×104).