[Paper Review] Scalable Bayesian Optimization Using Deep Neural Networks

Jasper Snoek, Oren Rippel+7 more|arXiv (Cornell University)|Feb 19, 2015

Machine Learning and Data Classification60 references439 citations

TL;DR

This paper proposes Deep Networks for Global Optimization (DNGO), a scalable Bayesian optimization method that replaces Gaussian processes with deep neural networks to model the posterior distribution over functions. By using neural networks for adaptive basis function regression, DNGO achieves linear scaling with data size—enabling massively parallel hyperparameter tuning—and finds state-of-the-art performance on CIFAR-10 (6.37% error) and CIFAR-100 (27.4% error) with minimal sequential steps.

ABSTRACT

Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization. In this work, we explore the use of neural networks as an alternative to GPs to model distributions over functions. We show that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically. This allows us to achieve a previously intractable degree of parallelism, which we apply to large scale hyperparameter optimization, rapidly finding competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.

Motivation & Objective

• Address the cubic computational scaling of Gaussian processes in Bayesian optimization, which limits scalability in high-evaluation scenarios.
• Enable massively parallel hyperparameter optimization by replacing expensive GP inference with a scalable neural network surrogate.
• Develop a method that maintains uncertainty quantification and exploration-exploitation balance while scaling linearly with data.
• Demonstrate the effectiveness of the approach on large-scale machine learning problems, including deep neural network hyperparameter tuning.
• Create a robust, generalizable optimization framework that generalizes across diverse global optimization tasks without requiring problem-specific tuning of the tuner itself.

Proposed method

• Replace the Gaussian process prior with a deep neural network to model the posterior distribution over functions in Bayesian optimization.
• Use the neural network to learn an adaptive set of basis functions for Bayesian linear regression, enabling efficient function approximation.
• Train the neural network surrogate using stochastic gradient descent, allowing scalable and incremental updates as new observations arrive.
• Employ a standard acquisition function (e.g., Expected Improvement) over the neural network's predictive mean and variance to guide the next evaluation point.
• Leverage the model’s uncertainty estimates to balance exploration and exploitation during optimization, similar to GP-based methods.
• Apply the method to large-scale hyperparameter search by distributing evaluations across parallel computing resources, such as Intel Xeon Phi coprocessors and high-performance clusters.

Experimental results

Research questions

• RQ1 Can deep neural networks effectively replace Gaussian processes as the surrogate model in Bayesian optimization while maintaining uncertainty quantification?
• RQ2 Does the proposed DNGO method achieve linear scaling with respect to the number of function evaluations, enabling large-scale parallel optimization?
• RQ3 Can DNGO discover hyperparameter configurations that match or exceed state-of-the-art performance on complex deep learning tasks like image classification?
• RQ4 How does the performance of DNGO compare to GP-based Bayesian optimization in terms of convergence speed and final solution quality on large-scale benchmarks?
• RQ5 To what extent can DNGO generalize across different machine learning problems without requiring re-tuning of the optimization framework itself?

Key findings

• DNGO achieves linear scaling with the number of observations, in contrast to the cubic scaling of standard Gaussian processes, enabling scalable optimization.
• The method successfully discovers a convolutional neural network configuration that attains a test error of 6.37% on CIFAR-10, matching state-of-the-art performance.
• On CIFAR-100, DNGO finds a model with a test error of 27.4%, significantly outperforming baseline methods and achieving competitive results.
• The approach enables efficient, massively parallel hyperparameter tuning, with 40 jobs running in parallel on Intel Xeon Phi coprocessors.
• The final model trained on the full CIFAR-100 dataset for 350 epochs using DNGO-optimized hyperparameters achieves a test error of 27.4%, demonstrating strong generalization.
• DNGO achieves competitive results on image caption generation, attaining BLEU scores of 25.1 (single model) and 26.7 (ensemble) on the Microsoft COCO 2014 dataset.