[Paper Review] Stochastic Distributed Optimization for Machine Learning from Decentralized Features

Yaochen Hu, Di Niu+2 more|arXiv (Cornell University)|Dec 16, 2018

Privacy-Preserving Technologies in Data31 references4 citations

TL;DR

This paper proposes FDML, a novel asynchronous stochastic gradient descent framework for distributed machine learning where features of the same samples are held across multiple parties without sharing raw data or model parameters. By exchanging only local predictions and leveraging a hyper-linear aggregation structure, FDML achieves performance close to centralized training while preserving data locality and enabling differential privacy, significantly improving app recommendation in real-world deployment at Tencent.

ABSTRACT

Distributed machine learning has been widely studied in the literature to scale up machine learning model training in the presence of an ever-increasing amount of data. We study distributed machine learning from another perspective, where the information about the training same samples are inherently decentralized and located on different parities. We propose an asynchronous stochastic gradient descent (SGD) algorithm for such a feature distributed machine learning (FDML) problem, to jointly learn from decentralized features, with theoretical convergence guarantees under bounded asynchrony. Our algorithm does not require sharing the original feature data or even local model parameters between parties, thus preserving a high level of data confidentiality. We implement our algorithm for FDML in a parameter server architecture. We compare our system with fully centralized training (which violates data locality requirements) and training only based on local features, through extensive experiments performed on a large amount of data from a real-world application, involving 5 million samples and $8700$ features in total. Experimental results have demonstrated the effectiveness and efficiency of the proposed FDML system.

Motivation & Objective

• To address the challenge of training a joint machine learning model when features of the same samples are distributed across multiple parties that cannot share raw data or model parameters.
• To preserve data locality and enhance privacy by avoiding transmission of original features or local model parameters between parties.
• To develop a scalable and efficient system that outperforms models trained on isolated local features while approaching the performance of centralized training.
• To integrate differential privacy into the communication protocol to further protect sensitive information during model collaboration.
• To evaluate the framework on real-world datasets from Tencent apps, demonstrating practical utility in large-scale recommendation systems.

Proposed method

• The FDML system uses a parameter server architecture to coordinate asynchronous updates, where each party trains its local model independently on its own feature set.
• Each party computes a local prediction for its mini-batch of samples and communicates only this prediction to the central server, not raw features or model parameters.
• A hyper-linear structure (similar to softmax) aggregates local predictions into a final joint prediction for classification tasks.
• The system employs mini-batched stochastic gradient descent in a stale synchronous parallel (SSP) setting, allowing bounded asynchrony with theoretical convergence guarantees.
• Differential privacy is applied by adding calibrated noise to the shared local predictions to protect against inference attacks.
• The framework supports arbitrary models (e.g., logistic regression, factorization machines, deep neural networks) and enables end-to-end training of the joint model.

Experimental results

Research questions

• RQ1 Can a collaborative machine learning model be trained effectively when features of the same samples are distributed across multiple parties without sharing raw data?
• RQ2 Can the joint model trained via FDML achieve performance comparable to centralized training while preserving data locality?
• RQ3 How does the system perform under bounded asynchrony, and what is its theoretical convergence rate?
• RQ4 To what extent can differential privacy be integrated into the communication protocol without degrading model performance?
• RQ5 How does FDML scale in terms of communication overhead and training efficiency on large-scale, real-world datasets?

Key findings

• FDML significantly outperforms models trained only on local features, achieving higher AUC and lower log loss on both the a9a dataset and Tencent’s 5M-record dataset.
• For logistic regression, FDML performance closely matches centralized training, with only a minor gap due to limited feature interaction in the hyper-linear structure.
• For deep neural networks, FDML achieves substantially better performance than local models and approaches centralized performance, though with a slight gap due to the absence of direct cross-party feature interaction in the architecture.
• In the NN setting, FDML is faster than centralized training due to distributed computation, especially when model size is large and communication overhead is manageable.
• Adding differential privacy with noise levels up to 3 still yields better performance than local-only learning, demonstrating strong privacy-utility trade-offs.
• On smaller datasets like a9a, communication overhead dominates, making FDML slower than centralized training, but this effect diminishes on larger, real-world datasets.