[Paper Review] Global optimization via inverse distance weighting.

Alberto Bemporad|arXiv (Cornell University)|Jun 15, 2019

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

This paper proposes a computationally efficient global optimization algorithm that combines inverse distance weighting (IDW) and radial basis functions (RBFs) to construct a surrogate model and guide acquisition sampling. It achieves performance comparable or superior to Bayesian optimization on benchmark problems, with support for simple constraints and open-source MATLAB/Python implementations.

ABSTRACT

Global optimization problems whose objective function is expensive to evaluate can be solved effectively by recursively fitting a surrogate function to function samples and minimizing an acquisition function to generate new samples. The acquisition step trades off between seeking for a new optimization vector where the surrogate is minimum (exploitation of the surrogate) and looking for regions of the feasible space that have not yet been visited and that may potentially contain better values of the objective function (exploration of the feasible space). This paper proposes a new global optimization algorithm that uses a combination of inverse distance weighting (IDW) and radial basis functions (RBF) to construct the acquisition function. Rather arbitrary constraints that are simple to evaluate can be easily taken into account by the approach. Compared to Bayesian optimization, the proposed algorithm is computationally lighter and, as we show in a set of benchmark global optimization and hyperparameter tuning problems, it has a very similar (and sometimes superior) performance. MATLAB and Python implementations of the proposed approach are available at http://cse.lab.imtlucca.it/~bemporad/idwgopt

Motivation & Objective

• Address the challenge of expensive-to-evaluate objective functions in global optimization, where each function evaluation is costly in time or resources.
• Develop a lightweight alternative to Bayesian optimization that maintains high performance while reducing computational overhead.
• Enable easy integration of simple, inexpensive constraints into the optimization process.
• Provide a practical and efficient algorithm suitable for real-world applications such as hyperparameter tuning and engineering design.

Proposed method

• Construct a surrogate model using inverse distance weighting (IDW) to interpolate function values at sampled points, favoring closer samples with higher influence.
• Combine IDW with radial basis functions (RBFs) to form a smooth surrogate that captures local and global trends in the objective function.
• Define an acquisition function that balances exploitation (minimizing the surrogate) and exploration (targeting unvisited regions of the feasible space).
• Iteratively select new sampling points by minimizing the acquisition function, ensuring a dynamic trade-off between exploration and exploitation.
• Incorporate constraints by evaluating them directly during candidate selection, without modifying the surrogate model.
• Use a trust-region-like strategy to ensure sampling stability and convergence, guided by the surrogate and acquisition function.

Experimental results

Research questions

• RQ1 Can a surrogate-based optimization method using IDW and RBFs outperform or match the performance of Bayesian optimization in global optimization tasks?
• RQ2 How does the proposed method handle constraints that are simple to evaluate but may restrict the feasible region?
• RQ3 To what extent does the computational cost of the proposed algorithm remain low compared to Bayesian optimization?
• RQ4 Does the method maintain robust performance across diverse benchmark problems and hyperparameter tuning scenarios?

Key findings

• The proposed IDW-RBF algorithm achieves performance on par with or better than Bayesian optimization across a range of global optimization benchmarks.
• The algorithm demonstrates superior or comparable convergence speed and accuracy in hyperparameter tuning tasks, particularly in high-dimensional settings.
• The method is computationally lighter than Bayesian optimization due to the absence of probabilistic inference and marginal likelihood optimization.
• Constraints that are easy to evaluate can be seamlessly integrated into the sampling process without altering the surrogate model structure.
• The open-source MATLAB and Python implementations enable reproducibility and practical deployment in research and industrial applications.
• The approach maintains strong exploration-exploitation balance through a well-designed acquisition function built on IDW-RBF interpolation.