[Paper Review] A Framework for Distributed Deep Learning Layer Design in Python

Clay McLeod|arXiv (Cornell University)|Oct 25, 2015

Computational Physics and Python Applications1 references3 citations

TL;DR

This paper presents a full-stack Python framework for distributed training and systematic evaluation of Deep Neural Networks (DNNs), leveraging Docker, Celery, RabbitMQ, and Theano to enable scalable, modular, and reproducible hyperparameter experimentation. It demonstrates that DNN performance plateaus beyond 500–700 hidden layers, suggesting a critical mass beyond which overfitting occurs.

ABSTRACT

In this paper, a framework for testing Deep Neural Network (DNN) design in Python is presented. First, big data, machine learning (ML), and Artificial Neural Networks (ANNs) are discussed to familiarize the reader with the importance of such a system. Next, the benefits and detriments of implementing such a system in Python are presented. Lastly, the specifics of the system are explained, and some experimental results are presented to prove the effectiveness of the system.

Motivation & Objective

• To enable systematic, large-scale experimentation on DNN hyperparameters and layer architectures to uncover empirical design rules.
• To provide a simple, modular, and extensible distributed system for training DNNs using only Python, appealing to researchers with minimal systems programming experience.
• To overcome limitations of Python’s Global Interpreter Lock (GIL) by using process-based parallelism via Celery and Docker containers for true CPU parallelism.
• To create a production-ready, distributed pipeline that supports high-throughput training, centralized result storage in MongoDB, and real-time monitoring via web dashboards.

Proposed method

• The system uses Docker containers to encapsulate the training environment, ensuring reproducibility and portability across machines.
• Celery task queues manage distributed training jobs, with workers processing tasks in parallel across multiple CPU cores or machines.
• RabbitMQ handles message passing between the task scheduler and workers, enabling reliable job distribution and monitoring.
• Theano is used as the deep learning backend with GPU acceleration, configured via THEANO_FLAGS for device and precision control.
• Results are stored in a MongoDB database with a RESTful API, enabling real-time access and visualization through plot.ly and web dashboards.
• The framework supports asynchronous job submission and includes monitoring dashboards for Celery and RabbitMQ to track worker status and job throughput.

Experimental results

Research questions

• RQ1 What is the impact of varying the number of hidden layers on DNN training time and model accuracy across different datasets?
• RQ2 How does the performance of DNNs scale with increasing depth, and does a critical layer count exist beyond which performance plateaus?
• RQ3 To what extent can a full-stack Python-based system achieve efficient distributed training despite the Global Interpreter Lock (GIL)?
• RQ4 Can a modular, composable framework built entirely in Python support high-throughput, scalable DNN hyperparameter search with minimal system complexity?
• RQ5 How effective is the integration of Docker, Celery, RabbitMQ, and MongoDB in enabling reproducible, distributed, and observable DNN experimentation?

Key findings

• DNN performance tends to plateau beyond 500–700 hidden layers, suggesting a critical mass point where further depth does not improve accuracy and may lead to overfitting.
• Training time increases approximately linearly with the number of hidden layers across a wide range of layer configurations.
• The system successfully mitigates the GIL bottleneck by using process-based parallelism via Celery, enabling efficient CPU utilization across multiple cores.
• The integration of Docker, Celery, and MongoDB enables a scalable, reproducible, and observable distributed training pipeline with low-latency result storage and real-time monitoring.
• The framework supports high-throughput training of 10,000–50,000 DNN configurations on a single machine or cluster, with results stored and visualized via RESTful APIs and web dashboards.
• Granular control over activation functions and hyperparameters leads to significant performance differences, highlighting the importance of systematic parameter search.