[Paper Review] Nonmyopic Gaussian Process Optimization with Macro-Actions

Dmitrii Kharkovskii, Chun Kai Ling+1 more|arXiv (Cornell University)|Feb 22, 2020

Advanced Bandit Algorithms Research26 references4 citations

TL;DR

This paper proposes a nonmyopic Gaussian process optimization framework using macro-actions to enable long-horizon planning in Bayesian optimization, overcoming the limitations of myopic and batch-based approaches. By introducing an $\epsilon$-Bayes-optimal macro-action policy and an anytime variant with theoretical performance guarantees, it achieves scalable, adaptive, and theoretically sound optimization for complex, high-budget objective functions.

ABSTRACT

This paper presents a multi-staged approach to nonmyopic adaptive Gaussian process optimization (GPO) for Bayesian optimization (BO) of unknown, highly complex objective functions that, in contrast to existing nonmyopic adaptive BO algorithms, exploits the notion of macro-actions for scaling up to a further lookahead to match up to a larger available budget. To achieve this, we generalize GP upper confidence bound to a new acquisition function defined w.r.t. a nonmyopic adaptive macro-action policy, which is intractable to be optimized exactly due to an uncountable set of candidate outputs. The contribution of our work here is thus to derive a nonmyopic adaptive epsilon-Bayes-optimal macro-action GPO (epsilon-Macro-GPO) policy. To perform nonmyopic adaptive BO in real time, we then propose an asymptotically optimal anytime variant of our epsilon-Macro-GPO policy with a performance guarantee. We empirically evaluate the performance of our epsilon-Macro-GPO policy and its anytime variant in BO with synthetic and real-world datasets.

Motivation & Objective

• To address the suboptimality of myopic and batch Bayesian optimization in high-budget, complex objective function settings.
• To enable nonmyopic, adaptive planning over extended horizons (up to budget size) by exploiting macro-actions as sequences of primitive actions.
• To develop a theoretically grounded, $\epsilon$-Bayes-optimal macro-action policy for Gaussian process optimization.
• To design an anytime variant of the policy with asymptotic optimality and performance guarantees for real-time deployment.

Proposed method

• Generalizes the GP upper confidence bound acquisition function to a nonmyopic adaptive macro-action policy, enabling lookahead planning.
• Introduces a novel $\epsilon$-Bayes-optimal macro-action GPO policy ($\epsilon$-Macro-GPO) to handle the intractability of optimizing over an uncountable set of macro-action outputs.
• Employs a recursive Bellman-style recursion to compute lookahead values, leveraging a structured macro-action policy with bounded complexity.
• Derives theoretical bounds on the number of macro-actions $N$ required to achieve $\epsilon$-optimality, showing $N = \mathcal{O}(\kappa^{2H}/\epsilon^2 \log(\kappa A / \epsilon))$.
• Proposes an anytime variant of $\epsilon$-Macro-GPO that maintains performance guarantees while enabling real-time execution.
• Uses auxiliary lemmas to bound the value function and propagation of uncertainty through macro-action sequences, ensuring theoretical tractability.

Experimental results

Research questions

• RQ1 Can macro-actions be effectively used to scale nonmyopic Bayesian optimization to longer horizons without sacrificing theoretical guarantees?
• RQ2 How can an $\epsilon$-Bayes-optimal policy be constructed for nonmyopic adaptive GPO when the macro-action output space is uncountable?
• RQ3 What is the theoretical sample complexity required for achieving $\epsilon$-optimality in macro-action-based nonmyopic GPO?
• RQ4 How can the proposed policy be made suitable for real-time deployment while preserving performance guarantees?
• RQ5 Does the proposed method outperform myopic and batch-based BO algorithms in terms of convergence to the global optimum under finite budget constraints?

Key findings

• The proposed $\epsilon$-Macro-GPO policy achieves $\epsilon$-Bayes-optimality for nonmyopic adaptive Gaussian process optimization with a theoretical sample complexity of $N = \mathcal{O}(\kappa^{2H}/\epsilon^2 \log(\kappa A / \epsilon))$.
• The anytime variant of $\epsilon$-Macro-GPO maintains asymptotic optimality and performance guarantees, enabling real-time deployment.
• Empirical evaluation on synthetic and real-world datasets shows that $\epsilon$-Macro-GPO outperforms myopic DB-GP-UCB and other baseline BO methods in finding the global maximum, especially under larger budgets.
• In a simulated AUV task, $\epsilon$-Macro-GPO successfully plans macro-actions to reach the global maximum hotspot, while myopic DB-GP-UCB converges to a local maximum.
• The method demonstrates scalability to lookaheads of up to 8 observations (H=4, N=1), significantly outperforming existing nonmyopic adaptive BO methods limited to lookaheads of 5 or fewer.
• Theoretical analysis confirms that the value function and uncertainty propagation are bounded via recursive lemmas, ensuring stability and convergence of the policy.