[Paper Review] Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settings

Sang Truong, Duc Quang Nguyen+5 more|arXiv (Cornell University)|Jan 10, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

LookaHES is a nonmyopic Bayesian optimization framework with dynamic, history-dependent costs that uses neural policies and pathwise sampling to plan long horizons (20+ steps), outperforming myopic and some nonmyopic baselines on synthetic and real tasks.

ABSTRACT

Bayesian optimization (BO) is a common framework for optimizing black-box functions, yet most existing methods assume static query costs and rely on myopic acquisition strategies. We introduce LookaHES, a nonmyopic BO framework designed for dynamic, history-dependent cost environments, where evaluation costs vary with prior actions, such as travel distance in spatial tasks or edit distance in sequence design. LookaHES combines a multi-step variant of $H$-Entropy Search with pathwise sampling and neural policy optimization, enabling long-horizon planning beyond twenty steps without the exponential complexity of existing nonmyopic methods. The key innovation is the integration of neural policies, including large language models, to effectively navigate structured, combinatorial action spaces such as protein sequences. These policies amortize lookahead planning and can be integrated with domain-specific constraints during rollout. Empirically, LookaHES outperforms strong myopic and nonmyopic baselines across nine synthetic benchmarks from two to eight dimensions and two real-world tasks: geospatial optimization using NASA night-light imagery and protein sequence design with constrained token-level edits. In short, LookaHES provides a general, scalable, and cost-aware solution for robust long-horizon optimization in complex decision spaces, which makes it a useful tool for researchers in machine learning, statistics, and applied domains. Our implementation is available at https://github.com/sangttruong/nonmyopia.

Motivation & Objective

• Address dynamic, history-dependent evaluation costs in Bayesian optimization.
• Enable scalable long-horizon planning beyond traditional four-step lookaheads.
• Incorporate neural policies to optimize decision variables and handle structured action spaces.
• Demonstrate effectiveness across synthetic benchmarks and real-world domains (protein design, geospatial optimization).
• Provide an implementation and empirical evaluation showing robustness to noise and cost structures.

Proposed method

• Formulate nonmyopic BO with dynamic cost using H-Entropy Search (HES) expanded to multiple steps (EHIG).
• Introduce LookaHES, combining a multi-step HES with pathwise sampling and neural policy optimization to reduce optimization parameters and trajectory samples.
• Model dynamic costs (Markovian and non-Markovian) and incorporate a Lagrangian budget term in the acquisition objective.
• Use a recurrent neural policy (RNN/transformer-based) to variationally optimize lookahead decisions, reducing horizon-dependent parameter growth.
• Apply pathwise sampling to generate fantasized trajectories and reduce rollout complexity from exponential to manageable levels.
• Handle discrete and continuous spaces by embedding tokens and employing suitable differentiation methods (reparameterization or policy gradients).

Experimental results

Research questions

• RQ1 RQ1: How does LookaHES compare to state-of-the-art myopic and nonmyopic baselines under dynamic costs in continuous inputs?
• RQ2 RQ2: Can LookaHES be effectively applied to problems with discrete input spaces?
• RQ3 RQ3: How do aleatoric/epistemic noises, surrogate model quality, and lookahead horizon affect LookaHES performance?
• RQ4 RQ4: Does optimism in myopic methods translate to better performance than nonmyopic methods, and can this optimism generalize to real-world problems?

Key findings

• LookaHES consistently outperforms myopic baselines across multiple cost structures and is competitive with nonmyopic baselines on synthetic benchmarks.
• The approach scales to horizons of twenty steps or more, addressing exponential rollout complexity via neural policy optimization and pathwise sampling.
• LookaHES demonstrates superior performance in discrete protein sequence design, leveraging a language-model-based policy to guide edits under spotlight costs.
• In continuous NASA night-light geospatial optimization, LookaHES shows robust improvements over baselines under dynamic costs.
• Using a LLaMa-3.2-based policy in protein editing, LookaHES achieves higher fluorescence with constrained edits and lower cumulative regret compared to myopic methods.
• The methodology is demonstrated across nine synthetic benchmarks (2D–8D) and two real-world tasks, illustrating generality and scalability.