[Paper Review] Open Loop Hyperparameter Optimization and Determinantal Point Processes

Jesse Dodge, Kevin Jamieson+1 more|arXiv (Cornell University)|Jun 6, 2017

Advanced Multi-Objective Optimization Algorithms25 references3 citations

TL;DR

This paper proposes using k-determinantal point processes (k-DPPs) for open-loop hyperparameter optimization to improve diversity in hyperparameter sampling compared to uniform random search. By leveraging a novel Metropolis-Hastings algorithm, k-DPPs are adapted to mixed discrete-continuous and structured search spaces, achieving superior performance over uniform sampling and even outperforming sequential Bayesian optimization in accuracy with significantly reduced wall-clock time.

ABSTRACT

Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions. In particular, we propose the use of $k$-determinantal point processes in hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a $k$-DPP promotes diversity. We describe an approach that transforms hyperparameter search spaces for efficient use with a $k$-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from $k$-DPPs defined over any space from which uniform samples can be drawn, including spaces with a mixture of discrete and continuous dimensions or tree structure. Our experiments show significant benefits in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

Motivation & Objective

• To address the inefficiency of uniform random search in hyperparameter optimization by promoting diversity in sampled configurations.
• To enable open-loop hyperparameter optimization with parallelizable, non-i.i.d. sampling that improves convergence and performance.
• To develop a general-purpose sampling method for k-DPPs over complex search spaces, including mixed discrete-continuous and tree-structured dimensions.
• To demonstrate that diversity-driven open-loop methods can surpass even state-of-the-art sequential Bayesian optimization in practical settings.

Proposed method

• Proposes k-determinantal point processes (k-DPPs) as a replacement for independent uniform sampling in open-loop hyperparameter search to encourage diversity.
• Introduces a novel Metropolis-Hastings algorithm that enables sampling from k-DPPs over arbitrary spaces, including mixed discrete-continuous and tree-structured search spaces.
• Transforms hyperparameter search spaces to make them compatible with k-DPPs, ensuring effective diversity promotion across heterogeneous dimensions.
• Employs a kernel-based k-DPP (k-DPP-RBF) using radial basis function kernels to model similarity between hyperparameter configurations.
• Uses a transformation-based approach to map complex search spaces into a form suitable for k-DPP sampling while preserving diversity properties.
• Employs a sequential evaluation framework where all configurations are pre-specified, enabling full parallelization and eliminating feedback dependency.

Experimental results

Research questions

• RQ1 Can k-DPP-based sampling improve hyperparameter optimization performance compared to uniform random search in both synthetic and real-world settings?
• RQ2 How does diversity in hyperparameter sampling affect convergence speed and final model accuracy under limited evaluation budgets?
• RQ3 To what extent can k-DPPs outperform sequential Bayesian optimization methods like Spearmint in terms of accuracy and wall-clock time?
• RQ4 Can k-DPPs be effectively applied to mixed-dimensional and structured hyperparameter spaces, such as those with discrete, continuous, and tree-structured components?
• RQ5 Does promoting diversity in open-loop search reduce the risk of getting stuck in local optima compared to adaptive closed-loop methods?

Key findings

• k-DPP-RBF sampling significantly outperformed uniform random search and Sobol sequences in terms of best-found model accuracy across all tested hyperparameter search spaces.
• In the 'Stable' search space, k-DPP-RBF achieved an average best accuracy of 82.61% with k=20, outperforming batch Spearmint (82.65%) and sequential Spearmint (82.76%) despite using only 1/10th of the wall-clock time.
• Sequential Bayesian optimization (Spearmint) required over ten times longer on average to find a solution, gaining only 0.15% accuracy improvement over k-DPP-RBF.
• The k-DPP method demonstrated consistent superiority in both synthetic and real-world experiments, especially in high-impact hyperparameter regimes.
• Even when all hyperparameters were within a 'stable' range, k-DPP-RBF still outperformed other open-loop methods, indicating that diversity improves performance even in well-behaved search spaces.
• The Metropolis-Hastings algorithm enabled effective k-DPP sampling over complex search spaces, validating the method’s generality and scalability.