[Paper Review] Black-box Optimization on Multiple Simplex Constrained Blocks

Priyam Das|arXiv (Cornell University)|Sep 8, 2016

Metaheuristic Optimization Algorithms Research14 references3 citations

TL;DR

This paper proposes Multi-Block GCDVSS, an efficient black-box optimization algorithm for objective functions with parameters constrained to multiple simplex blocks of varying sizes. By enabling simultaneous, adaptive step-size updates within each simplex block, it achieves up to 7× faster convergence than genetic algorithms while requiring only linear parallelization growth with dimensionality, significantly improving computational efficiency in high-dimensional settings.

ABSTRACT

Black-box optimization of objective function of parameters belonging to simplex arises in many inference and predictive models. Das (2016) introduced Greedy Co-ordinate Descent of Varying Step-sizes on Simplex (GCDVSS) which efficiently optimizes any black-box function whose parameters belong to a simplex. In this paper, that method has been modified and extended for the case where the set of parameters may belong to multiple simplex block of different sizes. The main principle of this algorithm is to make jumps of varying step-sizes within each simplexes simultaneously and searching for the best direction for movement. Since this algorithm is designed specially for multiple simplex blocks parameter space, the proportion of movements made within the parameter space during the update step of a iteration is relatively higher for the proposed algorithm. Starting from a single initial guess, unlike genetic algorithm or simulated annealing, requirement of parallelization for this algorithm grows linearly with the dimension of the parameter space which makes it more efficient for higher dimensional optimization problems. Comparative studies with some existing algorithms have been provided based on modified well-known benchmark functions. Upto 7 folds of improvement in computation time has been noted for using the proposed algorithm over Genetic algorithm, yielding significantly better solution in all the cases considered.

Motivation & Objective

• Address the need for efficient optimization in high-dimensional parameter spaces where parameters are constrained to multiple simplex blocks of different sizes.
• Overcome limitations of existing metaheuristics like genetic algorithms and simulated annealing, which scale poorly with dimensionality and require excessive parallelization.
• Develop a method that maintains high proportion of effective parameter space exploration per iteration by leveraging block-wise, adaptive step-size updates within each simplex.
• Enable efficient optimization of black-box functions in inference and predictive modeling where simplex constraints are common.
• Improve computational efficiency and solution quality compared to state-of-the-art methods in benchmark evaluations.

Proposed method

• Extends the Greedy Coordinate Descent with Varying Step-sizes (GCDVSS) algorithm to handle multiple simplex blocks instead of a single simplex.
• Performs simultaneous, adaptive step-size updates within each individual simplex block during each iteration to explore the parameter space efficiently.
• Uses a greedy strategy to select the best movement direction within each simplex block based on function evaluation feedback.
• Maintains feasibility by ensuring all parameter updates remain within the bounds of their respective simplex constraints.
• Employs a single initial guess and avoids population-based search, reducing memory and parallelization overhead.
• Scales parallelization linearly with the total dimension of the parameter space, unlike population-based methods that scale quadratically or worse.

Experimental results

Research questions

• RQ1 Can GCDVSS be effectively extended to handle multiple simplex blocks of varying sizes while preserving its efficiency and convergence properties?
• RQ2 How does the proposed Multi-Block GCDVSS algorithm compare in convergence speed and solution quality to established metaheuristic algorithms like genetic algorithms on constrained black-box optimization problems?
• RQ3 To what extent does the algorithm’s linear parallelization scaling improve its scalability in high-dimensional parameter spaces compared to population-based methods?
• RQ4 Does the block-wise, adaptive step-size strategy lead to a higher proportion of effective parameter space exploration per iteration than existing methods?
• RQ5 Can the algorithm achieve significant computational time improvements without sacrificing solution quality on benchmark functions with simplex constraints?

Key findings

• The proposed Multi-Block GCDVSS algorithm achieves up to 7-fold improvement in computation time compared to genetic algorithms across all tested benchmark functions.
• The algorithm consistently yields better solution quality than genetic algorithms in all test cases, demonstrating superior convergence performance.
• Parallelization requirements grow linearly with the dimension of the parameter space, making the method significantly more scalable than population-based alternatives.
• The proportion of effective movement within the parameter space during each update step is substantially higher than in traditional metaheuristics, due to block-wise, adaptive step-size selection.
• The method maintains feasibility and efficiency even in high-dimensional settings, outperforming existing approaches in both speed and solution accuracy.
• Empirical results on modified benchmark functions confirm the robustness and efficiency of the algorithm across diverse problem configurations with multiple simplex blocks.