[Paper Review] Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search

Linnan Wang, Rodrigo Fonseca+1 more|arXiv (Cornell University)|Jul 1, 2020

Advanced Bandit Algorithms Research55 references31 citations

TL;DR

LA-MCTS is a meta-algorithm that learns nonlinear space partitions for black-box optimization and uses BO within selected regions, improving sample efficiency especially in high-dimensional problems.

ABSTRACT

High dimensional black-box optimization has broad applications but remains a challenging problem to solve. Given a set of samples $\{\vx_i, y_i\}$, building a global model (like Bayesian Optimization (BO)) suffers from the curse of dimensionality in the high-dimensional search space, while a greedy search may lead to sub-optimality. By recursively splitting the search space into regions with high/low function values, recent works like LaNAS shows good performance in Neural Architecture Search (NAS), reducing the sample complexity empirically. In this paper, we coin LA-MCTS that extends LaNAS to other domains. Unlike previous approaches, LA-MCTS learns the partition of the search space using a few samples and their function values in an online fashion. While LaNAS uses linear partition and performs uniform sampling in each region, our LA-MCTS adopts a nonlinear decision boundary and learns a local model to pick good candidates. If the nonlinear partition function and the local model fits well with ground-truth black-box function, then good partitions and candidates can be reached with much fewer samples. LA-MCTS serves as a \emph{meta-algorithm} by using existing black-box optimizers (e.g., BO, TuRBO) as its local models, achieving strong performance in general black-box optimization and reinforcement learning benchmarks, in particular for high-dimensional problems.

Motivation & Objective

• Address high-dimensional black-box optimization by avoiding over-exploration through learned space partitioning.
• Leverage a hierarchical partitioning strategy to concentrate sampling in promising regions.
• Integrate a local optimizer (e.g., TuRBO or BO) within learned partitions to improve sample efficiency.
• Demonstrate effectiveness across MuJoCo RL benchmarks and synthetic functions.
• Provide ablations to understand the impact of hyper-parameters and partitioning strategy.

Proposed method

• Construct a tree where each node represents a region of the search space and recursively partition it via latent actions learned from current samples.
• Within a node, cluster samples by their function values using K-means to identify good vs. bad regions, then train an SVM to form a nonlinear boundary (latent action) for splitting.
• Use UCB to select a path through the partitioned space to balance exploration and exploitation, focusing samples on promising leaves.
• On the selected region, optimize f with a local solver (TuRBO or BO) while constraining search to the region defined by the path of latent-action boundaries.
• Iteratively split leaves when sample counts exceed a threshold to deepen the search in promising regions, refining v* estimates over time.
• Provide a sampling strategy inside constrained regions (Omega_selected) and adapt TuRBO initialization to the region, ensuring robust exploration.

Experimental results

Research questions

• RQ1 Can learned nonlinear space partitions improve sample efficiency in high-dimensional black-box optimization compared to fixed-partition methods?
• RQ2 Does integrating a local Bayesian optimizer within adaptively learned regions outperform standalone BO or EA methods?
• RQ3 How do hyper-parameters (C_p, kernel choice, splitting threshold) affect LA-MCTS performance across tasks?
• RQ4 Is LA-MCTS robust across diverse domains (MuJoCo RL tasks, trajectory optimization, synthetic benchmarks) as a general meta-optimizer?

Key findings

• LA-MCTS consistently outperforms state-of-the-art baselines (TuRBO, HesBO, BOHB, CMA-ES, DE, DOO, SOO, VOO) on MuJoCo locomotion tasks, especially as dimensionality increases.
• The learned nonlinear partitions and UCB-driven node selection enable focused exploration, reducing over-exploration and improving sample efficiency in high-dimensional problems.
• When paired with TuRBO, LA-MCTS substantially enhances performance and provides better initialization from promising regions across restarts.
• LA-MCTS also boosts gradient-based Bayesian optimization in high dimensions and improves performance on synthetic functions like Ackley and Rosenbrock.
• Ablation studies show exploration is crucial (Cp), kernel choice (SVM), and splitting threshold (theta) significantly impact results; deeper trees with appropriate thresholds help in large spaces.