[Paper Review] Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start

Valerio Perrone, Rodolphe Jenatton+2 more|arXiv (Cornell University)|Dec 8, 2017

Gaussian Processes and Bayesian Inference21 references15 citations

TL;DR

This paper proposes multiple adaptive Bayesian linear regression (ABL-R) with a shared neural network to enable scalable Bayesian optimization (BO) and transfer learning across multiple tasks. By jointly learning a deep feature representation and task-specific linear models, the method achieves linear scaling with data size—unlike cubic-scaling Gaussian processes—while enabling warm-start BO using prior evaluations from related tasks, significantly accelerating convergence in hyperparameter optimization.

ABSTRACT

Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization. Typically, BO is powered by a Gaussian process (GP), whose algorithmic complexity is cubic in the number of evaluations. Hence, GP-based BO cannot leverage large amounts of past or related function evaluations, for example, to warm start the BO procedure. We develop a multiple adaptive Bayesian linear regression model as a scalable alternative whose complexity is linear in the number of observations. The multiple Bayesian linear regression models are coupled through a shared feedforward neural network, which learns a joint representation and transfers knowledge across machine learning problems.

Motivation & Objective

• To address the scalability bottleneck of Gaussian process-based Bayesian optimization, which scales cubically with data size and limits reuse of past evaluations.
• To enable effective transfer learning in Bayesian optimization by leveraging knowledge from multiple related tasks, such as different datasets or machine learning workflows.
• To develop a model that maintains the uncertainty quantification of Bayesian methods while scaling linearly with the number of observations, suitable for large-scale deployment.
• To jointly optimize the feature representation (via a neural network) and task-specific Bayesian linear models, avoiding the need for separate pre-training steps.

Proposed method

• The method uses a shared feedforward neural network to learn a deep, task-invariant feature representation from inputs across multiple tasks.
• Each task is modeled via a separate Bayesian linear regression (BLR) with weights and precision parameters, conditioned on the shared neural network's output.
• The joint objective function maximizes the sum of log-marginal likelihoods across all tasks, enabling end-to-end training with principled uncertainty propagation.
• Posterior inference is analytically tractable: predictive mean and variance are computed using closed-form Gaussian updates based on the observed data and model hyperparameters.
• The approach supports heterogeneous signals by modeling multiple outputs per task (e.g., validation accuracy, training time), with separate precision parameters for each signal.
• The method is compared to random Fourier features and standard GPs, demonstrating superior scalability and performance in transfer learning settings.

Experimental results

Research questions

• RQ1 Can a scalable Bayesian model be designed to support transfer learning in Bayesian optimization without incurring cubic computational costs?
• RQ2 How does joint learning of a deep feature representation and task-specific Bayesian linear models compare to separate pre-training and fine-tuning in terms of optimization efficiency?
• RQ3 To what extent can knowledge from related tasks (e.g., different datasets or signals) accelerate convergence in hyperparameter optimization?
• RQ4 Does incorporating multiple heterogeneous signals (e.g., accuracy and training time) improve the performance of Bayesian optimization compared to single-output models?

Key findings

• ABL-R with a neural network basis achieves faster convergence in hyperparameter optimization compared to plain ABLR or GP-based BO when warm-started with data from 29 related tasks.
• The method scales linearly with the number of observations, enabling application to large datasets with over 750,000 total evaluations, unlike GP-based approaches which become infeasible at scale.
• Using a neural network basis outperforms random Fourier features (RKS) with D=100, despite RKS being faster, indicating that learned representations are more effective for transfer learning.
• Including contextual information (e.g., dataset size) did not yield consistent improvements, so it was omitted in subsequent experiments for simplicity.
• In multi-signal tuning (e.g., validation accuracy, training time), incorporating side signals accelerated convergence in ABLR-based hyperparameter optimization on LIBSVM datasets.
• The joint training of the neural network and BLR models via log-marginal likelihood maximization outperforms two-step approaches that pre-train the network with squared loss before switching to BLR.