[Paper Review] Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools

Ruben Mayer, Hans‐Arno Jacobsen|arXiv (Cornell University)|Mar 27, 2019

Advanced Neural Network Applications189 references45 citations

TL;DR

A comprehensive survey of challenges, techniques, and tools for scalable deep learning on distributed infrastructures, including a comparison of 11 open-source frameworks and analysis of parallelization, scheduling, and data management.

ABSTRACT

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.

Motivation & Objective

• Assess the challenges in running scalable deep learning on distributed infrastructures (resources, data movement, elasticity).
• Classify and synthesize techniques for parallel DL training (data, model, pipeline, hybrid) and their trade-offs.
• Evaluate open-source DL frameworks and orchestration tools for practical scalability.
• Discuss data management and multi-tenancy issues in large DL deployments.
• Highlight open research challenges and future directions in DL systems.

Proposed method

• Survey and synthesis of existing literature on scalable DL systems.
• Classification of parallelization strategies (data, model, pipeline, hybrid) and their trade-offs.
• Analysis and comparison of 11 open-source DL frameworks and tools.
• Discussion of infrastructure components, hardware trends, and interconnects.
• Evaluation of scheduling, elasticity, and data management in multi-tenant environments.
• Outlook on future research challenges in DL systems.

Experimental results

Research questions

• RQ1 What are the main challenges in managing large-scale DL on distributed infrastructures?
• RQ2 What parallelization techniques (data, model, pipeline, hybrid) are used and how do they compare?
• RQ3 How do centralized and decentralized parameter synchronization approaches perform in practice?
• RQ4 Which open-source frameworks and tools implement scalable DL techniques, and how do they map to hardware/infrastructure?
• RQ5 What are the open research directions and future trends in scalable DL systems?

Key findings

• Deep learning scalability hinges on three dimensions: model size, training data volume, and hardware infrastructure availability.
• Hardware heterogeneity and large-scale warehouse infrastructures drive the need for specialized interconnects, optimizations, and tailored DL runtimes.
• Data, model, and pipeline parallelism (and hybrids like Mesh-TensorFlow) are core strategies to scale training across distributed resources.
• Parameter synchronization architectures (centralized parameter servers vs. decentralized all-reduce) present trade-offs in throughput, fault tolerance, and deployment complexity.
• There is broad adoption of open-source DL frameworks and tools that implement distributed DL techniques, with ongoing research on automated optimization and scheduling.
• Future DL systems research should focus on multi-tenancy, data management, and end-to-end scalability in shared clusters and cloud environments.