[Paper Review] ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training

Qinqing Zheng, Bor-Yiing Su+8 more|arXiv (Cornell University)|Mar 7, 2020

Stochastic Gradient Optimization Techniques30 references4 citations

TL;DR

ShadowSync is a distributed training framework that decouples synchronization from training by executing it in a background thread, significantly reducing synchronization overhead and enabling high-frequency synchronization without sacrificing training throughput. This approach achieves superior model quality and linear scalability in large-scale recommendation systems, outperforming traditional foreground synchronization methods.

ABSTRACT

Recommendation systems are often trained with a tremendous amount of data, and distributed training is the workhorse to shorten the training time. While the training throughput can be increased by simply adding more workers, it is also increasingly challenging to preserve the model quality. In this paper, we present \shadowsync, a distributed framework specifically tailored to modern scale recommendation system training. In contrast to previous works where synchronization happens as part of the training process, \shadowsync separates the synchronization from training and runs it in the background. Such isolation significantly reduces the synchronization overhead and increases the synchronization frequency, so that we are able to obtain both high throughput and excellent model quality when training at scale. The superiority of our procedure is confirmed by experiments on training deep neural networks for click-through-rate prediction tasks. Our framework is capable to express data parallelism and/or model parallelism, generic to host various types of synchronization algorithms, and readily applicable to large scale problems in other areas.

Motivation & Objective

• Address the challenge of maintaining high model quality while scaling distributed training for large-scale recommendation systems.
• Overcome the performance bottleneck caused by synchronous training phases that stall training threads.
• Enable both data and model parallelism in a unified framework to support massive models and datasets.
• Achieve high training throughput and high synchronization frequency simultaneously, which are typically conflicting in traditional systems.
• Provide a generic, extensible framework supporting various synchronization algorithms for diverse distributed training workloads.

Proposed method

• Decouple synchronization from the main training thread by running it in a dedicated background 'shadow' thread.
• Use a shadow thread to perform parameter averaging and model updates independently of the training computation.
• Support both data parallelism (across trainers) and model parallelism (across embedding parameters) within the same framework.
• Enable multiple synchronization strategies (e.g., S-MA, S-BMUF, S-EASGD) to be used concurrently and independently for different components.
• Integrate intra-trainer Hogwild-style parallelism using multiple worker threads per trainer to maximize memory bandwidth utilization.
• Design the system to be agnostic to the underlying synchronization algorithm, allowing plug-and-play integration of various methods.

Experimental results

Research questions

• RQ1 Can separating synchronization from training reduce communication and computation bottlenecks in large-scale distributed training?
• RQ2 Does background synchronization enable higher synchronization frequency without degrading training throughput?
• RQ3 Can the framework maintain or improve model quality compared to traditional foreground synchronization in large-scale recommendation systems?
• RQ4 How does the decoupled synchronization model scale with increasing numbers of trainers and data size?
• RQ5 To what extent can the framework support both data and model parallelism while maintaining high efficiency and low overhead?

Key findings

• ShadowSync achieves linear scaling of effective parameter updates per second (EPS) without being limited by synchronization overhead.
• The S-MA and S-BMUF variants under ShadowSync show comparable or better model quality than their foreground counterparts (FR-MA, FR-BMUF), especially in terms of training and evaluation negative log-likelihood (NE).
• S-BMUF with a larger elastic parameter α converges faster and achieves better performance than the default setting, validating the design flexibility of the framework.
• The framework supports high-throughput training with 24 worker threads per trainer, as EPS gains plateau beyond this point due to memory bandwidth saturation.
• ShadowSync enables high-frequency synchronization—e.g., 2.9 syncs per minute for S-MA with 24 trainers—without impacting training throughput.
• The decentralized variants (S-BMUF, S-MA) perform comparably to the centralized S-EASGD, demonstrating that shadow synchronization is effective across different synchronization topologies.