[Paper Review] Predictive Entropy Search for Efficient Global Optimization of Black-box Functions

José Miguel Hernández-Lobato, Matthew W. Hoffman+1 more|arXiv (Cornell University)|Jun 10, 2014

Advanced Bandit Algorithms Research29 references396 citations

TL;DR

This paper introduces Predictive Entropy Search (PES), a Bayesian optimization method that selects evaluation points to maximize expected information gain about the global maximum of a black-box function. By reformulating the intractable entropy-based acquisition function using predictive distribution entropy, PES achieves more accurate and efficient optimization than Entropy Search and Expected Improvement, especially in noisy or multimodal settings.

ABSTRACT

We propose a novel information-theoretic approach for Bayesian optimization called Predictive Entropy Search (PES). At each iteration, PES selects the next evaluation point that maximizes the expected information gained with respect to the global maximum. PES codifies this intractable acquisition function in terms of the expected reduction in the differential entropy of the predictive distribution. This reformulation allows PES to obtain approximations that are both more accurate and efficient than other alternatives such as Entropy Search (ES). Furthermore, PES can easily perform a fully Bayesian treatment of the model hyperparameters while ES cannot. We evaluate PES in both synthetic and real-world applications, including optimization problems in machine learning, finance, biotechnology, and robotics. We show that the increased accuracy of PES leads to significant gains in optimization performance.

Motivation & Objective

• Address the challenge of efficiently optimizing expensive-to-evaluate, black-box functions with unknown derivatives and noisy evaluations.
• Overcome limitations of existing acquisition functions like Expected Improvement (EI), which can be overly greedy and fail to explore adequately.
• Develop a principled information-theoretic approach that quantifies information gain about the location of the global maximum.
• Enable fully Bayesian treatment of model hyperparameters, which previous entropy-based methods like Entropy Search could not efficiently handle.
• Improve optimization performance on both synthetic and real-world problems, including machine learning hyperparameter tuning, robotics, and biotechnology.

Proposed method

• Propose a novel acquisition function based on the expected reduction in differential entropy of the predictive distribution over the function's global maximum.
• Reformulate the intractable information-theoretic objective by approximating the entropy of the posterior predictive distribution at candidate points.
• Use Monte Carlo sampling over hyperparameters and function realizations to estimate the expected information gain, enabling efficient optimization of the acquisition function.
• Integrate hyperparameter marginalization into the acquisition function via sampling, allowing full Bayesian treatment of GP hyperparameters.
• Leverage Gaussian process priors with a kernel function to model the latent function, with posterior updates based on observed noisy evaluations.
• Optimize the acquisition function sequentially using a trust-region algorithm, selecting the next evaluation point to maximize expected information gain about the global maximizer.

Experimental results

Research questions

• RQ1 Can an information-theoretic acquisition function that maximizes expected information gain about the global maximum outperform heuristic methods like Expected Improvement?
• RQ2 How can the intractable entropy-based acquisition function be efficiently approximated for practical Bayesian optimization?
• RQ3 Can the method support a fully Bayesian treatment of model hyperparameters, unlike previous entropy-based approaches?
• RQ4 Does the improved information-theoretic formulation lead to better optimization performance on complex, multimodal, or noisy objective functions?
• RQ5 How does PES compare to Entropy Search and Expected Improvement in terms of convergence speed and final regret across diverse synthetic and real-world problems?

Key findings

• PES achieves significantly lower immediate regret than Entropy Search (ES) and Expected Improvement (EI) on synthetic benchmark functions, especially in high-dimensional or multimodal settings.
• On the Hartmann function, PES outperforms EI and ES, demonstrating that its more aggressive exploration is beneficial in complex landscapes despite requiring more iterations.
• In real-world applications like neural network hyperparameter tuning (NNet), hydrogen production (Hydrogen), and portfolio optimization (Portfolio), PES performs comparably to ES and better than EI, which gets stuck due to excessive greediness.
• On the bipedal robot walking speed optimization (Walker A and B), PES outperforms both EI and ES, particularly in the noisier Walker B setting, where its fully Bayesian treatment of hyperparameters provides a key advantage.
• PES-NB, a variant with fixed hyperparameters, underperforms PES, confirming that marginalizing over hyperparameters is crucial for performance in noisy environments.
• The method's ability to accurately approximate the intractable entropy-based acquisition function leads to more robust and efficient optimization, especially in high-dimensional and non-convex problems.