[Paper Review] PIVODL: Privacy-preserving vertical federated learning over distributed labels

Hangyu Zhu, Rui Wang+2 more|arXiv (Cornell University)|Aug 25, 2021

Privacy-Preserving Technologies in Data4 citations

TL;DR

PIVODL proposes a privacy-preserving vertical federated learning framework for training XGBoost models when labels are distributed across multiple clients. It combines homomorphic encryption and partial differential privacy to prevent leakage of label information during gradient and leaf weight transmission, achieving negligible performance degradation and strong privacy guarantees even under inference attacks.

ABSTRACT

Federated learning (FL) is an emerging privacy preserving machine learning protocol that allows multiple devices to collaboratively train a shared global model without revealing their private local data. Non-parametric models like gradient boosting decision trees (GBDT) have been commonly used in FL for vertically partitioned data. However, all these studies assume that all the data labels are stored on only one client, which may be unrealistic for real-world applications. Therefore, in this work, we propose a secure vertical FL framework, named PIVODL, to train GBDT with data labels distributed on multiple devices. Both homomorphic encryption and differential privacy are adopted to prevent label information from being leaked through transmitted gradients and leaf values. Our experimental results show that both information leakage and model performance degradation of the proposed PIVODL are negligible.

Motivation & Objective

• Address the realistic scenario in vertical federated learning where data labels are distributed across multiple clients rather than centralized on one party.
• Mitigate privacy risks arising from gradient and leaf weight transmission when labels are split across clients.
• Prevent differential attacks that exploit differences in gradient and Hessian sums across feature splits.
• Protect label privacy during model inference by preventing clients from guessing true labels from leaf weight values.
• Develop a secure, efficient, and privacy-preserving framework for training gradient boosting decision trees in a distributed label setting.

Proposed method

• Introduce a secure node split protocol where source and split clients jointly compute impurity scores without revealing label information.
• Use additive homomorphic encryption (Paillier) to securely aggregate gradients and Hessians across clients during tree node splitting.
• Implement a partial differential privacy mechanism that adds Gaussian noise only to leaf weight values sent to source clients to prevent label inference.
• Decompose the XGBoost training process into client-specific computations for data splitting and gain score calculation, minimizing exposure of label-related data.
• Design a communication-efficient protocol that ensures only encrypted or perturbed intermediate values are shared, preserving data locality.
• Ensure that all sensitive information—gradients, Hessians, and leaf weights—is protected through cryptographic and privacy-preserving mechanisms.

Experimental results

Research questions

• RQ1 Can a secure vertical federated learning system be designed for XGBoost when labels are distributed across multiple clients?
• RQ2 How can gradient and Hessian aggregation be securely performed when labels are not available on a single client?
• RQ3 What mechanisms can prevent differential privacy attacks that exploit differences in gradient sums across feature splits?
• RQ4 Can differential privacy be applied selectively to leaf weights to prevent label inference without degrading model performance?
• RQ5 To what extent does the proposed system preserve model accuracy while ensuring label privacy under inference attacks?

Key findings

• The proposed PIVODL framework achieves negligible model performance degradation, with test performance relatively insensitive to varying levels of differential privacy (ε).
• Without differential privacy, guess accuracy for label inference attacks exceeds 60% on both the Credit card and Bank marketing datasets, indicating high risk of label leakage.
• After applying partial differential privacy with ε = 10, guess accuracy drops to 14.45% on the Bank marketing dataset and 27.78% on the Credit card dataset, falling below the 50% random guess threshold.
• The use of homomorphic encryption increases communication costs by a maximum of 25MB for a model with six boosting trees on the Appliance energy prediction dataset, which is considered manageable.
• Training time is significantly impacted by encryption, with homomorphic encryption contributing to a large portion of the total training time, indicating a need for optimization.
• The partial differential privacy mechanism effectively prevents clients from inferring true labels through leaf weight analysis, ensuring that no label information is leaked during inference.