[Paper Review] Verifiable Coded Computing: Towards Fast, Secure and Private Distributed Machine Learning.

Tingting Tang, Ramy E. Ali+4 more|arXiv (Cornell University)|Jul 27, 2021

Privacy-Preserving Technologies in Data31 references4 citations

TL;DR

This paper proposes Verifiable Coded Computing (VCC), a decoupled framework that uses coded computing for straggler mitigation and data privacy, while employing verifiable computing to detect Byzantine workers. VCC accelerates distributed logistic regression by 3.2×–6.9× and improves test accuracy by up to 12.6% compared to uncoded methods.

ABSTRACT

Stragglers, Byzantine workers, and data privacy are the main bottlenecks in distributed cloud computing. Several prior works proposed coded computing strategies to jointly address all three challenges. They require either a large number of workers, a significant communication cost or a significant computational complexity to tolerate malicious workers. Much of the overhead in prior schemes comes from the fact that they tightly couple coding for all three problems into a single framework. In this work, we propose Verifiable Coded Computing (VCC) framework that decouples Byzantine node detection challenge from the straggler tolerance. VCC leverages coded computing just for handling stragglers and privacy, and then uses an orthogonal approach of verifiable computing to tackle Byzantine nodes. Furthermore, VCC dynamically adapts its coding scheme to tradeoff straggler tolerance with Byzantine protection and vice-versa. We evaluate VCC on compute intensive distributed logistic regression application. Our experiments show that VCC speeds up the conventional uncoded implementation of distributed logistic regression by $3.2 imes-6.9 imes$, and also improves the test accuracy by up to $12.6\%$.

Motivation & Objective

• Address the limitations of prior coded computing schemes that tightly couple straggler mitigation, Byzantine fault tolerance, and data privacy, leading to high communication or computational overhead.
• Decouple Byzantine node detection from straggler handling to reduce system complexity and improve efficiency.
• Enable dynamic tradeoffs between straggler tolerance and Byzantine protection based on workload demands.
• Improve performance and accuracy in distributed machine learning by combining coded computing with verifiable computing.
• Achieve practical deployment in real-world distributed ML workloads such as logistic regression with minimal overhead.

Proposed method

• Decouple the coding framework: use coded computing solely for straggler mitigation and data privacy, while using verifiable computing for Byzantine node detection.
• Apply linear coding techniques to encode data and computations across workers to tolerate stragglers and preserve privacy.
• Integrate verifiable computing via cryptographic proofs to detect and isolate malicious or Byzantine workers without relying on coding schemes.
• Dynamically adjust the coding parameters to balance straggler tolerance and Byzantine protection based on system conditions.
• Use a two-phase computation model: first, workers perform coded computations; second, results are verified using proof-of-correctness mechanisms.
• Leverage orthogonal mechanisms—coding for stragglers and privacy, verifiable computing for Byzantine faults—to reduce system-wide complexity.

Experimental results

Research questions

• RQ1 Can decoupling straggler tolerance from Byzantine fault detection reduce system overhead in distributed machine learning?
• RQ2 How does the integration of verifiable computing improve security against malicious workers without increasing coding complexity?
• RQ3 To what extent can dynamic coding adaptation improve the tradeoff between straggler resilience and Byzantine protection?
• RQ4 What performance and accuracy gains does VCC achieve over conventional uncoded distributed training in logistic regression?
• RQ5 Can VCC maintain data privacy while ensuring correctness and low latency in real-world distributed ML workloads?

Key findings

• VCC achieves a 3.2× to 6.9× speedup over conventional uncoded distributed logistic regression implementations.
• The framework improves test accuracy by up to 12.6% compared to uncoded baselines, indicating enhanced model robustness.
• By decoupling coding and verification, VCC reduces the computational and communication overhead associated with prior monolithic coded computing schemes.
• Dynamic adaptation of coding parameters allows VCC to balance straggler tolerance and Byzantine protection based on workload requirements.
• The integration of verifiable computing enables reliable detection of Byzantine workers without compromising the efficiency of the coded computing layer.
• VCC demonstrates practical feasibility and performance gains in a real-world compute-intensive distributed machine learning workload.