[Paper Review] TBD: Benchmarking and Analyzing Deep Neural Network Training

Hongyu Zhu, Mohamed Akrout+5 more|arXiv (Cornell University)|Mar 16, 2018

Adversarial Robustness in Machine LearningComputer Science59 references55 citations

TL;DR

The paper proposes a new TBD benchmark suite for DNN training across diverse domains and frameworks, plus a memory-profiling toolchain, and analyzes performance across TensorFlow, MXNet, CNTK on various hardware configurations.

ABSTRACT

The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.

Motivation & Objective

• Motivate the need for a broad DNN training benchmark beyond inference and image classification.
• Define TBD as a representative suite covering multiple domains (image classification, translation, speech, object detection, adversarial nets, reinforcement learning).
• Develop an end-to-end performance analysis toolchain for DNN training across major frameworks and hardware configurations.
• Create memory profiling tools to quantify memory usage for weights, activations, gradients, and workspace in TensorFlow, MXNet, and CNTK.
• Provide findings and recommendations to guide future research and optimization in DNN training.

Proposed method

• Curate a broad benchmark suite with six domains and eight state-of-the-art models across TensorFlow, MXNet, and CNTK.
• Evaluate training performance on single-GPU, multi-GPU, and multi-machine setups.
• Construct an end-to-end analysis toolchain by integrating existing profilers with domain-specific metrics.
• Develop memory profilers for three major frameworks to attribute memory usage to weights, activations, gradients, and workspace.
• Normalize implementations across frameworks to ensure comparable hyper-parameters and network definitions.

Experimental results

Research questions

• RQ1 What are the primary bottlenecks in DNN training across different models, frameworks, and hardware configurations?
• RQ2 How does memory usage in training differ across data structures (weights, activations, gradients, workspace) and frameworks?
• RQ3 How do throughput and GPU utilization vary among frameworks (TensorFlow, MXNet, CNTK) for diverse training workloads?
• RQ4 What actionable recommendations emerge to improve DNN training performance and memory efficiency?

Key findings

• RNN training is 2–3x less efficiently utilized by GPUs compared to image-classification models.
• GPU memory is often underutilized; simply exhausting memory with large mini-batches yields limited benefits across many models.
• Feature maps consume 70–90% of total memory during training, contrasting with inference where weights dominate memory.
• New memory profiling tools reveal precise allocations for weights, gradients, feature maps, and workspace across frameworks.
• The TBD benchmark and tooling provide directions for optimizing applications, libraries, and hardware in DNN training.