[Paper Review] Dynamic Sparse Graph for Efficient Deep Learning

Liu Liu, Lei Deng+5 more|arXiv (Cornell University)|Oct 1, 2018

Advanced Neural Network Applications60 references4 citations

TL;DR

This paper proposes Dynamic Sparse Graph (DSG), a novel training and inference framework that dynamically activates only high-selectivity neurons via dimension-reduction search and double-mask selection to achieve significant memory and computation savings. DSG reduces memory usage by 1.7–4.5x and operations by 2.3–4.4x with minimal accuracy loss across multiple benchmarks, while maintaining batch normalization compatibility and enabling efficient deployment on edge devices.

ABSTRACT

We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimension-reduction search (DRS) and obtains the BN compatibility via a double-mask selection (DMS). Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.

Motivation & Objective

• To address the memory and computation bottleneck in DNN training, where activations—not weights—dominate memory usage.
• To overcome the limitations of existing sparsity methods that are optimized for inference but complicate training due to dynamic activations and BN incompatibility.
• To enable both training and inference acceleration through a unified, BN-compatible sparsity mechanism that preserves model accuracy.
• To reduce representational redundancy by activating only the most selective neurons at each training iteration.
• To support deployment on resource-constrained edge devices by minimizing memory footprint and computational cost during both training and inference.

Proposed method

• DSG employs a dimension-reduction search to identify and activate only the most selective neurons at each training iteration, reducing activation memory and computation.
• A double-mask selection mechanism preserves batch normalization (BN) compatibility by maintaining activation statistics across mini-batches despite sparsity.
• The method dynamically reconfigures the computational graph per iteration, ensuring sparsity is maintained throughout forward and backward passes.
• Sparsity is applied at the neuron level, not weight level, targeting activation redundancy rather than weight pruning.
• The framework supports both single-node and distributed training, with potential for hardware acceleration via tiling and data reuse optimization.
• After training, the same sparsity pattern can be reused for inference, enabling consistent efficiency across deployment phases.

Experimental results

Research questions

• RQ1 How can we effectively reduce memory and computation costs in DNN training without compromising model accuracy?
• RQ2 Why do existing inference-optimized sparsity techniques fail to accelerate training, and what are the key bottlenecks?
• RQ3 How can batch normalization be preserved in a sparse training setting, given that it inherently disrupts sparsity through activation reorganization?
• RQ4 Can dynamic, neuron-level sparsity be efficiently computed and maintained during backpropagation?
• RQ5 To what extent can dynamic sparsity reduce memory and compute costs across diverse DNN architectures?

Key findings

• DSG achieves up to 4.5x memory reduction during DNN training and 1.7x during inference, significantly reducing representational cost.
• The method reduces computational operations by up to 4.4x in inference and 2.3x in training, with minimal accuracy loss across multiple benchmarks.
• The double-mask selection mechanism successfully maintains batch normalization compatibility, preserving model accuracy despite sparsity.
• Experiments on VGG8, ResNet8, and AlexNet show that DSG outperforms smaller dense models with equivalent MACs in both training time and accuracy.
• The dynamic sparsity mechanism enables efficient forward and backward passes, making it suitable for both cloud and edge deployment.
• The approach is compatible with existing training pipelines and can be extended to distributed training scenarios.