[Paper Review] Privacy-Preserving Distributed Learning with Secret Gradient Descent.

Valentin Hartmann, Robert West|arXiv (Cornell University)|Jun 27, 2019

Privacy-Preserving Technologies in Data26 references6 citations

TL;DR

This paper proposes Secret Gradient Descent (SecGD), a privacy-preserving distributed learning method that enables clients to collaboratively train machine learning models without exposing their raw data. By adding and later removing temporary noise via secure message sharing and routing through anonymization networks like Tor, SecGD ensures that even a malicious server cannot link gradients to individual clients, achieving computational privacy under the hardness of the subset sum problem with only two honest clients.

ABSTRACT

In many important application domains of machine learning, data is a privacy-sensitive resource. In addition, due to the growing complexity of the models, single actors typically do not have sufficient data to train a model on their own. Motivated by these challenges, we propose Secret Gradient Descent (SecGD), a method for training machine learning models on data that is spread over different clients while preserving the privacy of the training data. We achieve this by letting each client add temporary noise to the information they send to the server during the training process. They also share this noise in separate messages with the server, which can then subtract it from the previously received values. By routing all data through an anonymization network such as Tor, we prevent the server from knowing which messages originate from the same client, which in turn allows us to show that breaking a client's privacy is computationally intractable as it would require solving a hard instance of the subset sum problem. This setup allows SecGD to work in the presence of only two honest clients and a malicious server, and without the need for peer-to-peer connections.

Motivation & Objective

• To address the challenge of training machine learning models on distributed, privacy-sensitive data without centralized data collection.
• To enable secure model training in a setting where only two clients are honest and the server is potentially malicious.
• To prevent the server from linking gradients to individual clients through cryptographic noise and anonymized communication.
• To eliminate the need for peer-to-peer connections while maintaining strong privacy guarantees.

Proposed method

• Each client adds temporary noise to their gradients before sending them to the server.
• Clients separately transmit the noise values to the server, enabling the server to reconstruct and subtract the noise from the received gradients.
• All communication is routed through an anonymization network like Tor to obscure the origin of messages.
• The system relies on the computational intractability of the subset sum problem to prevent privacy breaches.
• The server performs standard gradient descent using the cleaned gradients, while clients never reveal raw data.
• The protocol ensures that even with a malicious server, the probability of re-identifying a client’s data is computationally negligible.

Experimental results

Research questions

• RQ1 Can we train machine learning models collaboratively across distributed clients while preserving the privacy of individual training data?
• RQ2 Is it possible to achieve strong privacy guarantees with only two honest clients and a malicious server?
• RQ3 Can we prevent the server from linking gradients to specific clients without requiring peer-to-peer communication?
• RQ4 Does the use of noise obfuscation and anonymized routing lead to a computationally secure system?
• RQ5 Can the security of the protocol be reduced to a well-known hard problem like subset sum?

Key findings

• The protocol achieves information-theoretic privacy for client data under the assumption that the subset sum problem is computationally hard.
• Even with a malicious server, the probability of re-identifying a client’s data is negligible due to the cryptographic hardness of the underlying problem.
• The method operates securely with only two honest clients, making it practical for real-world deployment.
• Anonymization via Tor prevents the server from correlating messages to the same client, enhancing resistance to inference attacks.
• The protocol does not require peer-to-peer connections, simplifying deployment in constrained environments.
• The system maintains model accuracy comparable to standard distributed training, as noise is removed before gradient updates.