[Paper Review] PrivColl: Practical Privacy-Preserving Collaborative Machine Learning

Yanjun Zhang, Guangdong Bai+4 more|arXiv (Cornell University)|Jul 14, 2020

Privacy-Preserving Technologies in Data43 references4 citations

TL;DR

PrivColl is a lightweight, privacy-preserving collaborative machine learning framework that uses additive secret sharing to securely compute model updates via homomorphic addition, ensuring both data and model privacy. It achieves up to 216× speedup over MPC/HE-based methods for neural network training while maintaining model accuracy and provable privacy under honest-but-curious adversaries.

ABSTRACT

Collaborative learning enables two or more participants, each with their own training dataset, to collaboratively learn a joint model. It is desirable that the collaboration should not cause the disclosure of either the raw datasets of each individual owner or the local model parameters trained on them. This privacy-preservation requirement has been approached through differential privacy mechanisms, homomorphic encryption (HE) and secure multiparty computation (MPC), but existing attempts may either introduce the loss of model accuracy or imply significant computational and/or communicational overhead. In this work, we address this problem with the lightweight additive secret sharing technique. We propose PrivColl, a framework for protecting local data and local models while ensuring the correctness of training processes. PrivColl employs secret sharing technique for securely evaluating addition operations in a multiparty computation environment, and achieves practicability by employing only the homomorphic addition operations. We formally prove that it guarantees privacy preservation even though the majority (n-2 out of n) of participants are corrupted. With experiments on real-world datasets, we further demonstrate that PrivColl retains high efficiency. It achieves a speedup of more than 45X over the state-of-the-art MPC/HE based schemes for training linear/logistic regression, and 216X faster for training neural network.

Motivation & Objective

• Address the gap in practical privacy-preserving collaborative learning where existing cryptographic methods incur high computational or communication overhead.
• Preserve both raw training data and local model parameters from disclosure during collaborative training.
• Enable high-efficiency training of diverse models, including linear regression, logistic regression, and deep neural networks, under strong privacy guarantees.
• Achieve practicality by minimizing reliance on expensive operations like homomorphic multiplication and non-linear functions in cryptographic settings.
• Formally prove privacy preservation even when up to $ n-2 $ out of $ n $ participants are corrupted, under the honest-but-curious threat model.

Proposed method

• Employ a two-layer architecture with local data owners and an untrusted aggregation node, ensuring data and model locality.
• Use additive secret sharing to distribute and securely compute model updates using only homomorphic addition operations.
• Redesign the collaborative learning workflow to decompose gradient descent into additive primitives, avoiding costly homomorphic multiplication and non-linear operations.
• Apply chain rule decomposition in backpropagation to distribute computation across local nodes and the aggregator, preserving correctness.
• Ensure correctness for any gradient-based model, including linear/logistic regression and fully connected neural networks with sigmoid activations.
• Formally prove that the probability of an adversary learning sensitive data or model parameters is negligible ($ eq ext{negligible} $), even with $ n-2 $ corrupted parties.

Experimental results

Research questions

• RQ1 Can a privacy-preserving collaborative learning framework be designed that avoids the high computational and communication costs of HE and MPC while preserving strong privacy guarantees?
• RQ2 Is it possible to achieve end-to-end privacy for both training data and local model parameters using only homomorphic addition operations?
• RQ3 How does the performance of such a framework compare to state-of-the-art MPC/HE-based solutions across different machine learning models and network conditions?
• RQ4 Can the framework maintain model accuracy without introducing noise, unlike differential privacy-based approaches?
• RQ5 What is the theoretical privacy guarantee of the framework under an honest-but-curious adversary model with up to $ n-2 $ corrupted parties?

Key findings

• PrivColl achieves a speedup of over 45× compared to state-of-the-art MPC/HE-based schemes for training linear and logistic regression models.
• For neural network training, PrivColl is 216× faster than the MZ17 MPC-based approach, completing training in 18,367.88 seconds (5.1 hours) on a WAN with 60,000 samples.
• In the LAN setting, PrivColl trains a neural network with 60,000 samples in 1,352.87 seconds (~22.5 minutes), while MZ17 takes over 81 hours (294,239.7 seconds) even in LAN.
• The framework maintains model accuracy equivalent to non-private training, avoiding the accuracy degradation common in differential privacy methods.
• Communication overhead dominates in WAN settings, but computation remains efficient, with communication costs scaling sublinearly with data size.
• PrivColl's total overhead is dominated by communication in WAN settings, but remains practical—especially compared to MPC-based alternatives that are infeasible in WAN.