[Paper Review] Optuna: A Next-generation Hyperparameter Optimization Framework

Plambeck, Swantje, Schmidt, Maximilian+3 more|arXiv (Cornell University)|Jul 25, 2019

Machine Learning and Data ClassificationComputer Science23 references980 citations

TL;DR

Optuna is a next-generation hyperparameter optimization framework that introduces a define-by-run API for dynamic search space construction, efficient pruning and sampling algorithms, and a scalable, easy-to-deploy architecture. It outperforms existing frameworks in both efficiency and usability, demonstrating linear scalability with distributed workers and significant speedups via pruning in real-world applications like object detection, HPL benchmarking, and database tuning.

ABSTRACT

Hybrid systems, which combine both continuous and discrete behavior, are used in many fields, including robotics, biological systems, and control systems. However, due to their complexity, finding an accurate model is a challenge. This paper discusses the usage of symbolic regression to learn hybrid systems from data and specifically analyses learning parameters for a recent algorithm. Symbolic regression is a powerful tool that can automatically discover accurate and interpretable mathematical models in the form of symbolic expressions. Models generated by symbolic regression are a valuable tool for system identification and diagnosis, e.g., to predict future system behavior or detect anomalies. A major opportunity of our approach is the ability to detect transitions between different continuous behaviors of a system directly based on the dynamics. From a diagnosis perspective, this can advantageously be used to detect the system entering fault modes and identify their models. This paper presents a parameter study for a symbolic regression based identification algorithm.

Motivation & Objective

• Address the limitations of existing hyperparameter optimization frameworks that require static, pre-defined search spaces.
• Enable dynamic, runtime construction of complex hyperparameter spaces, especially for models with conditional or variable architectures.
• Improve optimization efficiency through integrated, customizable pruning and advanced sampling strategies like TPE and ASHA.
• Design a versatile, easy-to-deploy framework that supports both lightweight interactive experiments and large-scale distributed computing.
• Demonstrate the framework's effectiveness across diverse real-world applications beyond deep learning, including HPC and database tuning.

Proposed method

• Introduces a define-by-run API where hyperparameter search spaces are constructed dynamically during the objective function execution via a live trial object.
• Uses a study-based architecture where each optimization run is a Study, and each evaluation is a Trial, managed through a centralized study object.
• Employs Bayesian optimization with Tree-structured Parzen Estimator (TPE) and adaptive resource allocation via the ASHA pruning algorithm.
• Supports both sequential and distributed optimization using a client-server model that scales linearly with the number of workers.
• Enables user-defined conditional hyperparameters and dynamic architecture search (e.g., varying number of layers and units in neural networks) through method chaining on the trial object.
• Integrates pruning strategies that monitor intermediate results and terminate unpromising trials early, reducing resource waste.

Experimental results

Research questions

• RQ1 Can a define-by-run API significantly simplify the specification of complex, conditional hyperparameter spaces in neural architecture search?
• RQ2 How does dynamic search space construction impact usability and expressiveness compared to static configuration in existing frameworks?
• RQ3 To what extent can efficient pruning and sampling algorithms improve optimization efficiency under limited computational budgets?
• RQ4 Can a unified framework efficiently scale from lightweight interactive experiments to large-scale distributed hyperparameter tuning?
• RQ5 How does Optuna perform in real-world, non-deep learning workloads such as HPC benchmarks and database configuration?

Key findings

• Optuna’s define-by-run API allows dynamic construction of hyperparameter spaces, enabling complex, conditional architectures (e.g., variable-depth neural networks) with minimal code and no upfront configuration.
• The framework achieves linear scalability in distributed settings: test error remains stable across different numbers of workers, indicating efficient load distribution and minimal communication overhead.
• With pruning (e.g., ASHA), Optuna explored 937 parameter sets in 4 hours on RocksDB, compared to only 39 without pruning, reducing computation time from 372s to 30s on average.
• In the Open Images Object Detection 2018 competition, Optuna helped develop a model (PFDet) that achieved 2nd place, demonstrating real-world effectiveness in large-scale ML tasks.
• Optuna reduced the time to find optimal parameters in the High Performance Linpack (HPL) benchmark for a supercomputer, showing utility beyond deep learning.
• For FFmpeg encoding, Optuna discovered a parameter set with reconstruction error comparable to the second-best preset, proving effectiveness in non-ML systems tuning.