[Paper Review] Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Bin Gu, Guodong Liu+3 more|arXiv (Cornell University)|Feb 17, 2021

Machine Learning and Data Classification34 references4 citations

TL;DR

This paper proposes HOZOG, a novel hyperparameter optimization method that combines black-box and gradient-based approaches by using zeroth-order hyper-gradients to enable scalable, efficient, and flexible tuning of large-scale hyperparameters (up to 1,250). It achieves state-of-the-art performance in effectiveness, efficiency, scalability, simplicity, and flexibility (E2S2F) without requiring gradient computation through backpropagation, outperforming existing methods in convergence speed and applicability to non-smooth or complex functions.

ABSTRACT

Modern machine learning algorithms usually involve tuning multiple (from one to thousands) hyperparameters which play a pivotal role in terms of model generalizability. Black-box optimization and gradient-based algorithms are two dominant approaches to hyperparameter optimization while they have totally distinct advantages. How to design a new hyperparameter optimization technique inheriting all benefits from both approaches is still an open problem. To address this challenging problem, in this paper, we propose a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG). Specifically, we first exactly formulate hyperparameter optimization as an A-based constrained optimization problem, where A is a black-box optimization algorithm (such as deep neural network). Then, we use the average zeroth-order hyper-gradients to update hyperparameters. We provide the feasibility analysis of using HOZOG to achieve hyperparameter optimization. Finally, the experimental results on three representative hyperparameter (the size is from 1 to 1250) optimization tasks demonstrate the benefits of HOZOG in terms of simplicity, scalability, flexibility, effectiveness and efficiency compared with the state-of-the-art hyperparameter optimization methods.

Motivation & Objective

• To address the challenge of optimizing large-scale hyperparameters (up to 1,250) in modern machine learning models.
• To unify the strengths of black-box optimization (simplicity, flexibility) and gradient-based methods (efficiency, scalability) in hyperparameter tuning.
• To develop a method that is effective, efficient, scalable, simple, and flexible (E2S2F) for diverse learning algorithms and hyperparameter sizes.
• To enable hyperparameter optimization in non-smooth or complex settings where exact gradient computation fails.

Proposed method

• Formulates hyperparameter optimization as an A-based constrained optimization problem, where A is a black-box algorithm (e.g., deep neural network).
• Introduces zeroth-order hyper-gradients (ZOHG) to estimate hypergradient direction without requiring explicit gradient computation through backpropagation.
• Uses average zeroth-order hyper-gradients to update hyperparameters via gradient descent, enabling black-box-like flexibility with gradient-based efficiency.
• Establishes theoretical feasibility under Lipschitz continuity of the outer objective function.
• Employs a finite-difference approximation to estimate hypergradients in a black-box manner, avoiding the need for model-specific gradient implementations.
• Applies the method to bi-level optimization problems: minimizing validation error subject to optimal model training for fixed hyperparameters.

Experimental results

Research questions

• RQ1 Can a hyperparameter optimization method inherit the simplicity and flexibility of black-box optimization while achieving the efficiency and scalability of gradient-based methods?
• RQ2 Is it possible to achieve effective hyperparameter tuning for high-dimensional hyperparameter spaces (up to 1,250) without relying on exact gradient computation?
• RQ3 How does the performance of a zeroth-order hyper-gradient method compare to state-of-the-art black-box and gradient-based approaches in terms of convergence speed and generalization?
• RQ4 Can the proposed method be applied to non-smooth or discontinuous objective functions where traditional gradient methods fail?

Key findings

• HOZOG achieved faster convergence than REV (an exact gradient method) in the early training stages, completing one hyper-gradient computation in ~24 seconds compared to REV’s ~40 seconds for 1,250 hyperparameters.
• HOZOG outperformed random search (RS) and BOHB in suboptimality, gradient norm, and test error across all hyperparameter sizes (500 and 1,250).
• HOZOG demonstrated superior scalability in hyperparameter size (up to 1,250) where methods like GPBO, FOR, and HOAG failed due to implementation limitations.
• The method achieved strong performance on data hyper-cleaning tasks, showing robustness in non-smooth settings where exact gradient methods are inapplicable.
• HOZOG maintained high flexibility and simplicity, requiring no custom gradient code, unlike RFHO and other gradient-based methods.
• Theoretical analysis confirms feasibility under Lipschitz continuity, supporting convergence guarantees for continuous objective functions.