[Paper Review] SecureBoost: A Lossless Federated Learning Framework

Kewei Cheng, Tao Fan+5 more|arXiv (Cornell University)|Jan 25, 2019

Privacy-Preserving Technologies in Data37 references178 citations

TL;DR

SecureBoost presents a lossless, privacy-preserving gradient-boosting framework for vertically partitioned data in federated learning, achieving centralized-level accuracy without leaking private data.

ABSTRACT

The protection of user privacy is an important concern in machine learning, as evidenced by the rolling out of the General Data Protection Regulation (GDPR) in the European Union (EU) in May 2018. The GDPR is designed to give users more control over their personal data, which motivates us to explore machine learning frameworks for data sharing that do not violate user privacy. To meet this goal, in this paper, we propose a novel lossless privacy-preserving tree-boosting system known as SecureBoost in the setting of federated learning. SecureBoost first conducts entity alignment under a privacy-preserving protocol and then constructs boosting trees across multiple parties with a carefully designed encryption strategy. This federated learning system allows the learning process to be jointly conducted over multiple parties with common user samples but different feature sets, which corresponds to a vertically partitioned data set. An advantage of SecureBoost is that it provides the same level of accuracy as the non-privacy-preserving approach while at the same time, reveals no information of each private data provider. We show that the SecureBoost framework is as accurate as other non-federated gradient tree-boosting algorithms that require centralized data and thus it is highly scalable and practical for industrial applications such as credit risk analysis. To this end, we discuss information leakage during the protocol execution and propose ways to provably reduce it.

Motivation & Objective

• Define privacy-preserving machine learning over vertically partitioned data in a federated setting.
• Develop a lossless gradient-boosting framework that operates across multiple parties with common samples but different features.
• Propose secure data alignment and encrypted gradient aggregation to train trees without exposing private data.
• Analyze information leakage and discuss methods to provably reduce it while retaining accuracy.

Proposed method

• Formally define the vertical federated learning problem with an active party holding labels and passive parties holding features.
• Align data samples across parties under privacy constraints using a privacy-preserving protocol.
• Train a shared gradient-boosting model by encrypting gradient statistics (g_i, h_i) with Paillier encryption and aggregating them to find optimal splits.
• Use an active party to decrypt aggregated statistics to determine global splits while passive parties compute locally on encrypted data.
• Store split decision information and lookup tables at passive and active parties to enable secure prediction (inference).
• Prove losslessness by showing the federated model has identical loss to a centralized non-privacy-preserving model under same initialization and hyperparameters.

Experimental results

Research questions

• RQ1 How can vertically partitioned data be aligned privately across multiple parties in federated learning?
• RQ2 Can a gradient-boosting model be trained in a privacy-preserving, lossless manner across parties with encrypted gradient statistics?
• RQ3 What is the intrusion/leakage profile during training and inference, and how can it be reduced without sacrificing accuracy?
• RQ4 Does SecureBoost achieve accuracy comparable to centralized non-federated gradient-boosting methods?

Key findings

• The framework is lossless: SecureBoost matches the accuracy of centralized non-privacy-preserving models under identical initialization and hyper-parameters.
• Security analysis shows potential leakage, with active parties able to learn more about the instance space and split candidates; reduced-leakage variant (RL-SecureBoost) mitigates leakage.
• Experiments on two credit datasets (Credit 1 and Credit 2) show comparable performance to non-federated methods, with RL-SecureBoost preserving accuracy while reducing leakage.
• Scalability analyses indicate convergence curves similar to GBDT and XGBoost, and runtime scales roughly linearly with tree depth and data size.
• The framework has practical applicability to industrial tasks like credit risk analysis, and is implemented in the FATE project for federated learning.