[Paper Review] Private Outsourced Bayesian Optimization

Dmitrii Kharkovskii, Zhongxiang Dai+1 more|arXiv (Cornell University)|Oct 24, 2020

Gaussian Processes and Bayesian Inference9 citations

TL;DR

This paper introduces PO-GP-UCB, the first privacy-preserving Bayesian optimization algorithm for outsourced settings with provable regret guarantees. It uses random projection to transform sensitive data while preserving input distances, enabling a modeler to optimize using privatized data without compromising privacy or performance.

ABSTRACT

This paper presents the private-outsourced-Gaussian process-upper confidence bound (PO-GP-UCB) algorithm, which is the first algorithm for privacy-preserving Bayesian optimization (BO) in the outsourced setting with a provable performance guarantee. We consider the outsourced setting where the entity holding the dataset and the entity performing BO are represented by different parties, and the dataset cannot be released non-privately. For example, a hospital holds a dataset of sensitive medical records and outsources the BO task on this dataset to an industrial AI company. The key idea of our approach is to make the BO performance of our algorithm similar to that of non-private GP-UCB run using the original dataset, which is achieved by using a random projection-based transformation that preserves both privacy and the pairwise distances between inputs. Our main theoretical contribution is to show that a regret bound similar to that of the standard GP-UCB algorithm can be established for our PO-GP-UCB algorithm. We empirically evaluate the performance of our PO-GP-UCB algorithm with synthetic and real-world datasets.

Motivation & Objective

• Address the lack of privacy-preserving Bayesian optimization in outsourced settings where data curators and modelers are separate parties.
• Ensure that the performance of Bayesian optimization remains close to that of non-private GP-UCB when applied to privatized data.
• Provide a theoretical regret bound for the proposed algorithm under differential privacy constraints.
• Enable practical deployment in sensitive domains such as healthcare, finance, and real estate where data cannot be shared directly.
• Design a transformation that preserves both privacy and the geometric structure of input data to maintain optimization fidelity.

Proposed method

• Propose the private-outsourced-Gaussian process-upper confidence bound (PO-GP-UCB) algorithm for privacy-preserving Bayesian optimization in outsourced settings.
• Apply random projection to transform the original dataset into a privatized form that preserves pairwise distances between inputs.
• Use the Johnson-Lindenstrauss lemma to ensure that the projected data maintains sufficient structure for effective GP-UCB optimization.
• Integrate differential privacy via random projection to protect individual data points while maintaining utility.
• Establish theoretical regret bounds for PO-GP-UCB that match the form of standard GP-UCB under mild assumptions.
• Ensure that the modeler can perform optimization on the transformed data without access to the original dataset or full knowledge of the curator's data.

Experimental results

Research questions

• RQ1 Can a privacy-preserving Bayesian optimization algorithm be designed for the outsourced setting where the data curator and modeler are separate entities?
• RQ2 Does random projection preserve the necessary geometric structure of inputs to maintain GP-UCB performance under differential privacy?
• RQ3 Can a provable regret bound be established for a differentially private BO algorithm in the outsourced setting?
• RQ4 How does the performance of PO-GP-UCB compare to non-private GP-UCB when applied to privatized data?
• RQ5 What is the trade-off between privacy, data utility, and optimization efficiency in outsourced Bayesian optimization?

Key findings

• PO-GP-UCB achieves a regret bound that is asymptotically similar to that of the standard GP-UCB algorithm, even when applied to randomly projected, privatized data.
• The random projection-based transformation preserves the pairwise distances between inputs, ensuring that the kernel structure required for GP-UCB remains approximately intact.
• Empirical evaluation on synthetic and real-world datasets shows that PO-GP-UCB maintains competitive optimization performance compared to non-private GP-UCB.
• The algorithm ensures differential privacy by design, as the random projection acts as a privacy mechanism that limits inference about individual data points.
• Theoretical analysis confirms that the smallest eigenvalue of the inverse covariance matrix remains bounded under the transformation, supporting stable GP inference.
• The method is applicable to real-world scenarios such as medical record analysis, loan risk assessment, and property price optimization, where data privacy is critical.