[Paper Review] Accelerating Vertical Federated Learning

Dongqi Cai, Tao Fan+5 more|arXiv (Cornell University)|Jul 23, 2022

Privacy-Preserving Technologies in Data37 references4 citations

TL;DR

This paper proposes a straggler-resilient and computation-efficient system to accelerate vertical federated learning (VFL) under homomorphic encryption (HE). By combining a backup worker scheme to reduce communication overhead in heterogeneous environments and principal component analysis (PCA) for input matrix compression, the approach reduces communication overhead by up to 65.26% and computation overhead by up to 40.66%, significantly improving VFL efficiency without compromising security.

ABSTRACT

Privacy, security and data governance constraints rule out a brute force process in the integration of cross-silo data, which inherits the development of the Internet of Things. Federated learning is proposed to ensure that all parties can collaboratively complete the training task while the data is not out of the local. Vertical federated learning is a specialization of federated learning for distributed features. To preserve privacy, homomorphic encryption is applied to enable encrypted operations without decryption. Nevertheless, together with a robust security guarantee, homomorphic encryption brings extra communication and computation overhead. In this paper, we analyze the current bottlenecks of vertical federated learning under homomorphic encryption comprehensively and numerically. We propose a straggler-resilient and computation-efficient accelerating system that reduces the communication overhead in heterogeneous scenarios by 65.26% at most and reduces the computation overhead caused by homomorphic encryption by 40.66% at most. Our system can improve the robustness and efficiency of the current vertical federated learning framework without loss of security.

Motivation & Objective

• To address the high communication and computation overhead in vertical federated learning (VFL) under homomorphic encryption (HE), which hinders practical deployment.
• To improve robustness and efficiency in heterogeneous environments where worker speeds and data characteristics vary significantly.
• To reduce the computational burden of HE, which inflates data size and slows training by orders of magnitude.
• To maintain strong cryptographic security while accelerating training, ensuring privacy-preserving collaboration across institutions.
• To enable practical cross-industry collaboration in domains like finance and healthcare where data is siloed and privacy-critical.

Proposed method

• A straggler-resilient backup worker scheme is employed to mitigate communication delays caused by slow or unresponsive workers in heterogeneous VFL settings.
• The system uses principal component analysis (PCA) to compress input feature matrices before homomorphic encryption, reducing the size of data to be processed and encrypted.
• Communication overhead is minimized by allowing the aggregator to accept the first n gradients from n + b workers, discarding the remaining b as backups, thus avoiding waiting for all workers.
• The method preserves security by ensuring all intermediate results remain encrypted throughout training, relying on homomorphic encryption for privacy.
• The framework is implemented within the FATE (Federated AI Technology Enabler) open-source platform to ensure compatibility and real-world applicability.
• The approach is evaluated under realistic VFL scenarios with diverse data distributions and computational heterogeneity.

Figure 1 : Vertical federated learning. A specialization of federated learning for secure cross-silo cooperation learning.

Experimental results

Research questions

• RQ1 How can communication overhead in heterogeneous VFL systems be reduced without sacrificing security?
• RQ2 To what extent can input matrix compression via PCA reduce computation costs in HE-protected VFL?
• RQ3 Can a backup worker strategy effectively mitigate straggler effects in VFL, where intermediate results (not just gradients) are exchanged?
• RQ4 What is the trade-off between computational efficiency and model accuracy when applying PCA-based compression in encrypted VFL?
• RQ5 How can VFL be made more scalable and practical for real-world cross-silo collaboration in privacy-sensitive industries?

Key findings

• The proposed backup worker scheme reduces communication overhead in heterogeneous VFL scenarios by up to 65.26%.
• PCA-based input compression reduces computation overhead caused by homomorphic encryption by up to 40.66%.
• The system maintains strong cryptographic security by preserving end-to-end encryption throughout the training process.
• The acceleration is achieved without compromising model accuracy, as the system maintains training performance close to that of local training.
• The framework improves both robustness and efficiency, making VFL more viable for real-world deployment in finance, healthcare, and IoT applications.
• The approach is compatible with the FATE platform, enabling practical integration into existing industrial federated learning pipelines.

Figure 2 : The framework of vertical federated learning, where homomorphic encryption is applied to preserve privacy.