[Paper Review] Parallel Multi-Objective Hyperparameter Optimization with Uniform Normalization and Bounded Objectives

Romain Égelé, Tyler H. Chang+3 more|arXiv (Cornell University)|Sep 26, 2023

Advanced Multi-Objective Optimization AlgorithmsComputer Science3 citations

TL;DR

This paper proposes D-MoBO, a parallel multi-objective Bayesian optimization framework that uses uniform quantile normalization and a soft penalty to improve convergence and solution quality in hyperparameter tuning. It achieves a 5x speed-up with 16x more workers and outperforms NSGA-II and MoTPE on benchmark tasks, especially in early convergence and hypervolume indicator scores.

ABSTRACT

Machine learning (ML) methods offer a wide range of configurable hyperparameters that have a significant influence on their performance. While accuracy is a commonly used performance objective, in many settings, it is not sufficient. Optimizing the ML models with respect to multiple objectives such as accuracy, confidence, fairness, calibration, privacy, latency, and memory consumption is becoming crucial. To that end, hyperparameter optimization, the approach to systematically optimize the hyperparameters, which is already challenging for a single objective, is even more challenging for multiple objectives. In addition, the differences in objective scales, the failures, and the presence of outlier values in objectives make the problem even harder. We propose a multi-objective Bayesian optimization (MoBO) algorithm that addresses these problems through uniform objective normalization and randomized weights in scalarization. We increase the efficiency of our approach by imposing constraints on the objective to avoid exploring unnecessary configurations (e.g., insufficient accuracy). Finally, we leverage an approach to parallelize the MoBO which results in a 5x speed-up when using 16x more workers.

Motivation & Objective

• To address the challenges of multi-objective hyperparameter optimization (MOHPO), including objective scale disparities, outlier sensitivity, and inefficient exploration of uninteresting trade-offs.
• To improve iteration efficiency and solution diversity in model-based MOHPO, especially for expensive black-box functions common in ML workflows.
• To enable effective parallelization of MOHPO that scales with compute resources, avoiding performance saturation seen in Gaussian-process-based methods.
• To ensure only high-quality, feasible trade-offs are explored by incorporating a soft penalty for configurations outside a user-defined objective range.
• To outperform existing methods like NSGA-II and MoTPE in both convergence speed and final solution quality on real-world ML benchmarks.

Proposed method

• Uses quantile uniform normalization (QU) to rescale objectives, enhancing focus on the Pareto front and improving robustness to outliers.
• Applies a soft penalty function to discourage exploration of configurations with objective values outside a user-defined feasible range.
• Employs randomized scalarization via uniformly sampled weights to promote diversity in the search across the Pareto front.
• Utilizes a random forest surrogate model to efficiently model the multi-objective landscape and guide acquisition.
• Enables decentralized, parallel evaluation of multiple configurations using a shared queue and worker coordination, boosting scalability.
• Implements a bounded scalarization approach that restricts the search to only explore trade-offs within a specified range of interest.

Experimental results

Research questions

• RQ1 Can uniform quantile normalization improve the robustness and convergence of multi-objective Bayesian optimization in hyperparameter tuning?
• RQ2 How does a soft penalty function for infeasible or uninteresting objective trade-offs affect the efficiency and quality of MOHPO?
• RQ3 To what extent can parallelization improve the performance of model-based MOHPO, especially in terms of speed-up and solution quality?
• RQ4 How does D-MoBO compare to established baselines like NSGA-II and MoTPE in terms of hypervolume and convergence speed across diverse ML benchmarks?
• RQ5 Does the proposed method scale effectively with increasing numbers of parallel workers, and can it maintain performance gains without saturation?

Key findings

• D-MoBO achieved a 5x speed-up in convergence when using 16x more workers, reaching the solution of 40 workers in 30 minutes with 640 GPUs.
• With 640 workers, D-MoBO achieved a hypervolume indicator (HVI) of 0.66, significantly outperforming NSGA-II (0.66) and MoTPE (0.26) at the same scale.
• The area under the HVI curve (AUC) for D-MoBO with 640 workers was 0.51, surpassing NSGA-II (0.45), MoTPE (0.23), and Random search (0.13).
• The soft penalty mechanism reduced exploration of uninteresting configurations, improving efficiency—D-MoBO NP (no penalty) achieved lower HVI (0.51) than D-MoBO (0.66) with 640 workers.
• Random search showed minimal performance gain with increased parallelism, completing 30 evaluations at 640 workers but achieving only 0.21 HVI, indicating poor scalability.
• D-MoBO consistently outperformed all competitors across all worker counts, with the largest performance gap observed in early-stage convergence.