[论文解读] Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance

Shuhei Watanabe|arXiv (Cornell University)|Apr 21, 2023

Machine Learning and Data Classification被引用 122

一句话总结

本论文解析 Tree-Structured Parzen Estimator (TPE) 算法，分析每个控制参数的作用，并提出一个在多样基准测试上提升经验性能的推荐配置。

ABSTRACT

Recent scientific advances require complex experiment design, necessitating the meticulous tuning of many experiment parameters. Tree-structured Parzen estimator (TPE) is a widely used Bayesian optimization method in recent parameter tuning frameworks such as Hyperopt and Optuna. Despite its popularity, the roles of each control parameter in TPE and the algorithm intuition have not been discussed so far. The goal of this paper is to identify the roles of each control parameter and their impacts on parameter tuning based on the ablation studies using diverse benchmark datasets. The recommended setting concluded from the ablation studies is demonstrated to improve the performance of TPE. Our TPE implementation used in this paper is available at https://github.com/nabenabe0928/tpe/tree/single-opt.

研究动机与目标

• Explain the algorithm intuition behind TPE and its components.
• Identify how each control parameter affects exploration versus exploitation in TPE.
• Empirically evaluate ablations across diverse benchmarks to derive recommended settings.
• Compare the recommended settings with baseline methods to demonstrate performance gains.

提出的方法

• Model p(y|x, D) with KDEs by splitting observations into better and worse groups via a top-quantile gamma.
• Compute the density ratio r(x|D) = p(x|D(l)) / p(x|D(g)) as the acquisition function.
• Use a splitting algorithm (gamma) to control exploration vs exploitation (linear or sqrt variants).
• Apply a weighting algorithm to assign weights to KDE components, affecting the KDEs and acquisition.
• Incorporate bandwidth selection with a magic clipping mechanism to adapt exploration strength (b(min) and Scott’s rule).
• Optionally use a multivariate kernel to capture interactions between parameters, and include a non-informative prior p0 to stabilize early search.

实验结果

研究问题

• RQ1 What are the roles and impacts of each TPE control parameter on empirical performance?
• RQ2 How do choices like splitting gamma, weighting, bandwidth, and kernel type influence exploration versus exploitation?
• RQ3 Do the proposed default settings outperform baseline TPE configurations across a diverse suite of benchmarks?
• RQ4 How does incorporating a prior and multivariate kernels affect optimization in practice?

主要发现

• Ablation shows that component choices significantly influence exploration/exploitation balance and performance.
• Multivariate kernels can capture interactions and may outperform univariate kernels in many settings.
• Bandwidth modifications (magic clipping) crucially affect search precision versus exploration depending on noise levels.
• Priors help stabilize early search and prevent premature exploitation, especially with limited observations.
• Recommended settings derived from ablations improve TPE performance compared to baselines across diverse benchmarks.