[Paper Review] Hyperparameter optimization with approximate gradient

Fabián Pedregosa|arXiv (Cornell University)|Feb 7, 2016

Sparse and Compressive Sensing Techniques24 references139 citations

TL;DR

Presents Hoag, a gradient-based hyperparameter optimization algorithm that uses approximate gradients with summable error, ensuring convergence to a stationary point; demonstrated on regularization and kernel parameters.

ABSTRACT

Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.

Motivation & Objective

• Motivate the need for effective hyperparameter optimization in regularized and kernel-based models.
• Develop a gradient-based method that uses approximate gradients to reduce computational burden.
• Establish convergence guarantees under mild regularity and summability assumptions.
• Empirically evaluate Hoag on logistic regression and kernel ridge regression across multiple datasets.

Proposed method

• Formulate hyperparameter optimization as a bi-level problem with inner and outer objectives.
• Derive an approximate gradient for the outer objective using inexact solutions to the inner problem and a linear system.
• Define Hoag: solve inner problem up to tolerance εk, solve Hessian-linear system up to εk, form p_k from approximate gradient, perform projected gradient step with step size 1/L.
• Prove convergence: gradient error is O(εk) and summable εk implies convergence to a stationary point.
• Discuss adaptive step size and practical implementation details, including CG-based Hessian solves and εk-tolerance strategies.

Experimental results

Research questions

• RQ1 How can hyperparameter optimization be performed with approximate gradient information?
• RQ2 Under what conditions do approximate-gradient methods converge to stationary points in bi-level hyperparameter settings?
• RQ3 What are practical tolerance sequences and step-size strategies that yield competitive empirical performance?
• RQ4 How does Hoag compare with grid search, random search, SMBO, and iterative differentiation in accuracy and efficiency?
• RQ5 Can Hoag be effectively applied to regularization parameter estimation and kernel parameter tuning across datasets?

Key findings

• The gradient of the outer objective can be approximated with a controllable error that is O(εk).
• If the εk sequence is summable, Hoag converges to a stationary point of the outer objective.
• Hoag achieves competitive performance relative to grid search, random search, SMBO, and iterative differentiation on L2-regularized logistic regression and kernel Ridge regression.
• Adaptive step-size strategies can be used in practice to cope with unknown Lipschitz constants while maintaining convergence behavior.
• Hoag benefits from warm-starting inner optimizations, aiding efficiency.
• Empirical results suggest fast early convergence in some datasets despite non-monotonic overall progress.