[Paper Review] Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data

Timothy Castiglia, Anirban Das+2 more|arXiv (Cornell University)|Jun 16, 2022

Privacy-Preserving Technologies in Data5 citations

TL;DR

This paper proposes Compressed Vertical Federated Learning (C-VFL), a communication-efficient framework for vertically partitioned data where parties compress their model embeddings before sharing, reducing communication by over 90% without significant accuracy loss. It provides the first theoretical convergence analysis for embedding compression in VFL, proving $O(1/√{T})$ convergence under bounded compression error, and establishes conditions for convergence with quantization and top-$k$ sparsification.

ABSTRACT

We propose Compressed Vertical Federated Learning (C-VFL) for communication-efficient training on vertically partitioned data. In C-VFL, a server and multiple parties collaboratively train a model on their respective features utilizing several local iterations and sharing compressed intermediate results periodically. Our work provides the first theoretical analysis of the effect message compression has on distributed training over vertically partitioned data. We prove convergence of non-convex objectives at a rate of $O(\frac{1}{\sqrt{T}})$ when the compression error is bounded over the course of training. We provide specific requirements for convergence with common compression techniques, such as quantization and top-$k$ sparsification. Finally, we experimentally show compression can reduce communication by over $90\%$ without a significant decrease in accuracy over VFL without compression.

Motivation & Objective

• Address the high communication cost in Vertical Federated Learning (VFL) due to large embeddings in vertically partitioned data.
• Enable communication efficiency in VFL by compressing intermediate embeddings rather than gradients, which is novel compared to existing gradient compression in Horizontal FL.
• Provide the first theoretical analysis of convergence for VFL with embedding compression, establishing conditions under which convergence is preserved.
• Ensure practical applicability by deriving specific compression parameter bounds for common techniques like quantization and top-$k$ sparsification.
• Demonstrate empirically that compression reduces communication by over 90% with negligible impact on model accuracy.

Proposed method

• C-VFL uses block-coordinate descent with multiple local iterations (Q > 1) to reduce global communication frequency.
• Parties compute local embeddings via neural networks and compress them using techniques like quantization or top-$k$ sparsification before sending to the server.
• The server aggregates compressed embeddings, computes the global loss, and sends back only the gradient with respect to each party’s embedding to reduce communication.
• Each party computes the local gradient using the chain rule: $\nabla_{m}F = \nabla_{\theta_{m}}h_{m} \cdot \nabla_{h_{m}}F$, enabling local model updates.
• The framework supports arbitrary server models and generalizes prior VFL methods by enabling compression at the embedding level.
• Two algorithm variants are used: Algorithm 1 for Q > 1 (multiple local steps), and Algorithm 2 for Q = 1 (single step with compressed embedding transfer per round).

Experimental results

Research questions

• RQ1 Can embedding compression in VFL achieve significant communication reduction without degrading model convergence or accuracy?
• RQ2 What theoretical conditions on compression error ensure convergence of the global model in VFL with compressed embeddings?
• RQ3 How do common compression techniques like quantization and top-$k$ sparsification affect convergence in VFL, and what parameter settings guarantee convergence?
• RQ4 Does using multiple local iterations (Q > 1) in C-VFL reduce total communication cost compared to single-iteration transfer, despite higher per-round cost?
• RQ5 Can C-VFL maintain high model accuracy while reducing communication by over 90% in real-world datasets like MIMIC-III?

Key findings

• C-VFL achieves over 90% reduction in communication cost on the MIMIC-III dataset compared to non-compressed VFL, with minimal accuracy drop.
• The communication cost for reaching a target F1-score of 0.4 dropped from 4517.59 MB (no compression) to 125.09 MB when using 3-bit vector quantization with Q=25.
• With Q=1, Algorithm 2 reduces per-round communication cost but requires more global rounds, resulting in higher total communication than Algorithm 1 for Q>1.
• Theoretical analysis proves convergence at rate $O(1/\sqrt{T})$ under bounded compression error, matching non-compressed VFL performance.
• For quantization, a 3-bit representation ensures convergence when the compression error is bounded; for top-$k$ sparsification, the sparsity level must be sufficiently high to maintain convergence.
• Empirical results show that using Q>1 significantly reduces total communication cost compared to Q=1, despite higher per-round cost, due to fewer global rounds needed.