[Paper Review] Stochastic Hyperparameter Optimization through Hypernetworks

Jonathan Lorraine, David Duvenaud|arXiv (Cornell University)|Feb 26, 2018

Machine Learning and Data Classification17 references87 citations

TL;DR

The paper proposes training a hypernetwork to map hyperparameters to approximately optimal weights, enabling joint stochastic optimization of weights and hyperparameters and avoiding retraining from scratch for each hyperparameter set.

ABSTRACT

Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of weights and hyperparameters. Our process trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our technique converges to locally optimal weights and hyperparameters for sufficiently large hypernetworks. We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.

Motivation & Objective

• Motivate hyperparameter tuning as a nested optimization problem and reduce it to a differentiable, joint optimization of weights and hyperparameters.
• Introduce a hypernetwork that outputs approximately optimal weights given hyperparameters to enable efficient hyperparameter optimization.
• Show convergence and practical feasibility of jointly training weights and hyperparameters across large hyperparameter spaces.
• Compare hypernetwork-based optimization to standard hyperparameter strategies and demonstrate effectiveness on thousands of hyperparameters.

Proposed method

• Train a neural network (hypernetwork) that takes hyperparameters as input and outputs approximately optimal weights wφ(λ).
• Update hypernetwork parameters φ via the chain rule using the training loss: ∂LTrain(wφ)/∂wφ · ∂wφ/∂φ.
• Optionally, optimize hyperparameters λ by differentiating the validation loss through the hypernetwork: ∂LValid(wφ(λ))/∂wφ(λ) · ∂wφ(λ)/∂λ.
• Propose joint optimization (Algorithm 3) where φ and λ are updated in tandem using stochastic gradients.
• Discuss a scalable variant with a linear hypernetwork and local best-response approximation (Algorithm 4).
• Compare to Bayesian optimization and unrolled optimization, highlighting advantages in scaling to thousands of hyperparameters.

Experimental results

Research questions

• RQ1 Can a hypernetwork learn a continuous best-response w*(λ) that maps hyperparameters to optimal weights?
• RQ2 Does joint optimization of hypernetwork weights and hyperparameters converge to a local optimum under mild assumptions?
• RQ3 Is gradient-based optimization via hypernetworks more scalable than traditional hyperparameter methods for high-dimensional hyperparameter spaces?
• RQ4 How does hyper-training compare to unrolled optimization and Bayesian optimization in efficiency and performance?
• RQ5 What are practical strategies (e.g., linear hypernetworks, local distributions p(λ|λ̂)) to scale to thousands of hyperparameters?

Key findings

• Hypernetworks can approximate a best-response function and yield validation performance close to the true optimum in small problems.
• Joint optimization (Algorithm 3) with locally trained, simple hypernetworks can converge faster than unrolled optimization in large hyperparameter settings.
• Hyper-training with thousands of hyperparameters outperforms standard gradient-free methods in convergence speed and scalability.
• Deeper networks can be optimized with hyper-training, maintaining or improving validation performance as layer count grows.
• A linear or factorized hypernetwork can suffice for local best-responses, reducing parameter burden while enabling gradient-based hyperparameter updates.