[Paper Review] On $\ell_p$-hyperparameter Learning via Bilevel Nonsmooth Optimization

Takayuki Okuno, Akiko Takeda+2 more|arXiv (Cornell University)|Jun 5, 2018

Sparse and Compressive Sensing Techniques37 references4 citations

TL;DR

This paper proposes the first theoretically guaranteed algorithm for hyperparameter selection in $β_p$-regularized learning via bilevel nonsmooth optimization, using a smoothing-type method to handle the nonconvex, nonsmooth lower-level problem. The approach achieves convergence to optimality conditions (SB-KKT) and outperforms Bayesian optimization and grid search in efficiency and scalability across multiple datasets.

ABSTRACT

We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0

Motivation & Objective

• To address the lack of theoretical algorithms for hyperparameter selection in nonconvex, nonsmooth $β_p$-regularized learning problems with $0 < p \leq 1$.
• To develop a bilevel optimization framework where the upper-level problem minimizes validation error and the lower-level problem solves an $β_p$-regularized learning problem.
• To establish new necessary optimality conditions—SB-KKT—for bilevel problems involving nonsmooth, nonconvex $β_p$-regularizers.
• To design a scalable, smoothing-based algorithm with global convergence guarantees for this bilevel problem.
• To empirically demonstrate superior performance compared to Bayesian optimization and grid search in hyperparameter tuning.

Proposed method

• Proposes a smoothing-type algorithm to handle the nonsmooth $β_p$-regularizer in the lower-level problem, enabling gradient-based optimization.
• Introduces SB-KKT (Smoothing-based KKT) conditions as new necessary optimality conditions for bilevel problems with nonsmooth, nonconvex lower-level objectives.
• Uses a modified Newton-type method (Algorithm B.2) to solve the smoothed lower-level subproblems without requiring line search.
• Applies a bilevel optimization framework where the upper-level objective is the validation error, and the lower-level is the $β_p$-regularized training objective.
• Employs a smoothing parameter $\mu$ that is gradually reduced to approximate the original nonsmooth problem.
• Derives convergence guarantees showing that the algorithm's iterates accumulate at points satisfying the SB-KKT conditions under mild assumptions.

Experimental results

Research questions

• RQ1 Can a theoretically grounded algorithm be developed for hyperparameter selection in $β_p$-regularized learning with $0 < p \leq 1$?
• RQ2 What are the necessary optimality conditions for bilevel problems involving nonsmooth, nonconvex $β_p$-regularizers?
• RQ3 Can a smoothing-based approach achieve global convergence for such bilevel problems?
• RQ4 How does the proposed algorithm compare in efficiency and accuracy to Bayesian optimization and grid search?
• RQ5 Is the algorithm scalable to problems with multiple hyperparameters and large datasets?

Key findings

• The proposed algorithm achieves convergence to points satisfying the newly derived SB-KKT conditions under mild assumptions, providing the first theoretical guarantee for bilevel $β_p$-hyperparameter learning.
• On the CpuSmall dataset with 13 hyperparameters, the algorithm found a solution with validation error $1.285 \times 10^5$ in just 0.66 seconds, outperforming Bayesian optimization which required 531.8 seconds to reach a worse value.
• For the Student dataset with 273 hyperparameters, the algorithm achieved a validation error of 0.771 in 1.45 seconds, while Bayesian optimization took 26.76 seconds to reach 18.324.
• On the Facebook dataset with $p=1$, the algorithm reached a validation error of 6.474 in 17.4 seconds, compared to Bayesian optimization’s 42.26 seconds to reach 6.476.
• The algorithm consistently found better or comparable solutions faster than Bayesian optimization and grid search, with no need for expensive line search procedures.
• Supplementary results show that the proposed algorithm often finds the best solution earlier than Bayesian optimization, even under extended time limits.