[Paper Review] Bayesian Optimization with Directionally Constrained Search

Yang Li, Yaqiang Yao|arXiv (Cornell University)|Jun 22, 2019

Advanced Multi-Objective Optimization Algorithms21 references4 citations

TL;DR

This paper proposes a Bayesian optimization method that enforces directional constraints on search directions to improve efficiency under strict evaluation budgets. By incorporating directional preferences derived from a surrogate model into the acquisition function, the approach reduces unnecessary local exploration and accelerates convergence, outperforming standard methods like Expected Improvement in both synthetic and real-world benchmarks.

ABSTRACT

Bayesian optimization offers a flexible framework to optimize an objective function that is expensive to be evaluated. A Bayesian optimizer iteratively queries the function values on its carefully selected points. Subsequently, it makes a sensible recommendation about where the optimum locates based on its accumulated knowledge. This procedure usually demands a long execution time. In practice, however, there often exists a computational budget or an evaluation limitation allocated to an optimizer, due to the resource scarcity. This constraint demands an optimizer to be aware of its remaining budget and able to spend it wisely, in order to return as better a point as possible. In this paper, we propose a Bayesian optimization approach in this evaluation-limited scenario. Our approach is based on constraining searching directions so as to dedicate the model capability to the most promising area. It could be viewed as a combination of local and global searching policies, which aims at reducing inefficient exploration in the local searching areas, thus making a searching policy more efficient. Experimental studies are conducted on both synthetic and real-world applications. The results demonstrate the superior performance of our newly proposed approach in searching for the optimum within a prescribed evaluation budget.

Motivation & Objective

• To address the inefficiency of standard Bayesian optimization under strict evaluation budgets, where excessive local exploration wastes computational resources.
• To reduce oscillatory behavior near the optimum by incorporating directional constraints into the search policy.
• To balance global exploration and local exploitation by dynamically adjusting search directions based on surrogate model uncertainty.
• To improve convergence speed and solution quality within a fixed number of function evaluations.

Proposed method

• The method introduces a directional constraint into the Expected Improvement acquisition function, favoring search directions aligned with the most promising region.
• It models the search direction as a von Mises-Fisher distribution, encoding directional preferences using concentration parameters and mean directions.
• The algorithm updates the directional distribution iteratively using a Bayesian update rule based on observed function values and gradient estimates from the surrogate model.
• A hierarchical model is constructed where the concentration parameter κ is updated using a Taylor approximation of the log-normalizing constant of the von Mises-Fisher distribution.
• The search direction is refined over time by combining prior directional beliefs with new evidence, ensuring the optimizer focuses on high-potential regions.
• The approach dynamically reduces exploration in unproductive directions, especially as the evaluation budget nears exhaustion.

Experimental results

Research questions

• RQ1 How can Bayesian optimization be made more efficient under strict evaluation budget constraints?
• RQ2 Can directional constraints improve convergence speed by reducing oscillatory behavior near the optimum?
• RQ3 How can directional preferences be encoded and updated in a Bayesian optimization framework without access to gradients?
• RQ4 What is the impact of incorporating directional constraints on the performance of Expected Improvement in black-box optimization?
• RQ5 Can a surrogate model's uncertainty be leveraged to guide search direction selection in the absence of gradient information?

Key findings

• The proposed method significantly reduces oscillatory behavior in the final stages of optimization, leading to more stable convergence.
• On synthetic functions, the method achieves a 25% improvement in convergence speed compared to standard Expected Improvement under a fixed budget of 50 evaluations.
• In real-world hyperparameter tuning tasks, the approach finds better-performing configurations with fewer evaluations than baseline BO methods.
• The directional constraint effectively reduces wasted evaluations in unproductive regions, especially during later iterations.
• The Bayesian update rule for the concentration parameter κ ensures adaptive refinement of search direction preferences over time.
• Empirical results confirm that the method maintains global exploration early on while focusing on high-potential areas as the budget depletes.