[Paper Review] Differentially Private Bayesian Learning on Distributed Data

Mikko Heikkilä, Eemil Lagerspetz+4 more|arXiv (Cornell University)|Mar 3, 2017

Privacy-Preserving Technologies in Data28 references15 citations

TL;DR

This paper proposes a differentially private Bayesian learning framework for distributed data, combining secure multi-party computation (DCA) with Gaussian mechanism-based noise injection to enable efficient, privacy-preserving inference. The method achieves near-optimal accuracy comparable to trusted aggregator settings while scaling efficiently with data size and party count, demonstrating practicality on linear regression tasks with minimal privacy cost.

ABSTRACT

Many applications of machine learning, for example in health care, would benefit from methods that can guarantee privacy of data subjects. Differential privacy (DP) has become established as a standard for protecting learning results. The standard DP algorithms require a single trusted party to have access to the entire data, which is a clear weakness. We consider DP Bayesian learning in a distributed setting, where each party only holds a single sample or a few samples of the data. We propose a learning strategy based on a secure multi-party sum function for aggregating summaries from data holders and the Gaussian mechanism for DP. Our method builds on an asymptotically optimal and practically efficient DP Bayesian inference with rapidly diminishing extra cost.

Motivation & Objective

• To address the limitation of existing differentially private (DP) machine learning methods that require a single trusted party with full data access.
• To enable DP Bayesian inference in a distributed setting where each client holds only a subset of data, preserving individual privacy.
• To minimize the extra noise introduced by distributed aggregation, ensuring high utility of the learned models.
• To develop a practical, efficient, and provably private framework that combines secure multi-party computation with DP Bayesian inference.

Proposed method

• Uses a secure multi-party computation (DCA) protocol to compute the sum of sufficient statistics across distributed clients without revealing individual data.
• Applies the Gaussian mechanism for (ϵ, δ)-differential privacy by injecting noise into the sum of sufficient statistics at the aggregator level.
• Employs secret sharing with M independent compute nodes to ensure that no single node can reconstruct individual contributions.
• Uses fixed-point arithmetic to enable exact cancellation of noise during sum aggregation, ensuring correct noise distribution.
• Introduces a private normalization procedure for data scaling in the distributed setting to maintain privacy guarantees.
• Combines the asymptotically optimal sufficient statistics perturbation (SSP) method with distributed noise injection to maintain statistical efficiency.

Experimental results

Research questions

• RQ1 Can differentially private Bayesian learning be effectively performed in a fully distributed setting without a trusted aggregator?
• RQ2 How can the noise variance be kept low in distributed DP learning while preserving privacy?
• RQ3 What is the performance overhead of using secure multi-party computation for DP sum aggregation in Bayesian inference?
• RQ4 How does the distributed DP method compare to the trusted aggregator baseline in terms of model accuracy and privacy budget efficiency?
• RQ5 Can private data normalization be achieved in a distributed setting without compromising the DP guarantee?

Key findings

• The distributed DP Bayesian learning method achieves predictive performance nearly indistinguishable from the trusted aggregator baseline, with no significant accuracy degradation.
• Runtime scales modestly with the number of clients and features, indicating good practical scalability on a Spark-based prototype.
• The extra noise introduced by the distributed setting is negligible in practice, with results effectively matching those of the trusted aggregator setting.
• Private data normalization is successfully achieved in the distributed setting, enabling the use of sufficient statistics perturbation with full privacy guarantees.
• The method maintains asymptotic optimality of the SSP approach, with convergence to non-private inference performance as the number of parties increases.
• The use of symmetric cryptography and fixed-point arithmetic enables efficient, low-overhead secure computation without requiring homomorphic encryption.