[Paper Review] Structured Deep Neural Network Pruning via Matrix Pivoting

Ranko Sredojević, Shaoyi Cheng+3 more|arXiv (Cornell University)|Dec 1, 2017

Neural Networks and Applications11 references8 citations

TL;DR

This paper proposes a structured pruning method for deep neural networks using matrix pivoting to create a permutation-block-permutation (PBP) form, enabling efficient sparse matrix-vector multiplication on edge GPUs. The approach achieves near-linear speedup with reduced coefficients, outperforming vendor-optimized libraries by improving memory bandwidth utilization and enabling hardware-aware optimization.

ABSTRACT

Deep Neural Networks (DNNs) are the key to the state-of-the-art machine vision, sensor fusion and audio/video signal processing. Unfortunately, their computation complexity and tight resource constraints on the Edge make them hard to leverage on mobile, embedded and IoT devices. Due to great diversity of Edge devices, DNN designers have to take into account the hardware platform and application requirements during network training. In this work we introduce pruning via matrix pivoting as a way to improve network pruning by compromising between the design flexibility of architecture-oblivious and performance efficiency of architecture-aware pruning, the two dominant techniques for obtaining resource-efficient DNNs. We also describe local and global network optimization techniques for efficient implementation of the resulting pruned networks. In combination, the proposed pruning and implementation result in close to linear speed up with the reduction of network coefficients during pruning.

Motivation & Objective

• Address the challenge of deploying computation-heavy deep neural networks (DNNs) on resource-constrained edge devices such as mobile and IoT systems.
• Overcome the inefficiency of generic sparse matrix libraries in handling moderate sparsity (5–25% fill-in) common in DNN pruning.
• Bridge the gap between architecture-agnostic pruning and architecture-aware optimization by introducing a structured sparsity model compatible with hardware execution patterns.
• Enable efficient, low-latency inference on modern embedded and mobile GPUs by optimizing both single-layer and multi-layer FC layers.
• Demonstrate that structured sparsity via matrix pivoting preserves model accuracy while significantly improving inference speed

Proposed method

• Introduce a permutation-block-permutation (PBP) matrix representation that reorganizes sparse weight matrices into blocks with structured sparsity, minimizing irregular memory access.
• Apply static matrix pivoting during pruning to transform arbitrary sparse weight matrices into PBP form, enabling efficient GPU execution.
• Design local optimization strategies for single-layer FC layers by leveraging the PBP structure to reduce memory access overhead.
• Develop multi-layer compiler optimizations that preserve PBP structure across consecutive FC layers, enabling cross-layer memory access coalescing.
• Implement a custom kernel for sparse matrix-vector multiplication (gemv) in PBP format, tuned for modern GPU architectures like NVIDIA Jetson TX1 and ARM Mali T880.
• Use vendor-optimized libraries (cuBLAS, cuSPARSE, Arm Compute Library) as baselines for performance comparison

Experimental results

Research questions

• RQ1 Can structured sparsity via matrix pivoting enable near-linear speedup in DNN inference on edge GPUs while maintaining high model accuracy?
• RQ2 How does the PBP representation improve memory bandwidth utilization compared to standard sparse formats like CSR or BRC?
• RQ3 To what extent does the permutation overhead in PBP transformation affect performance on small or highly sparse matrices?
• RQ4 Can the PBP-based pruning and optimization strategy outperform vendor-optimized dense and sparse linear algebra libraries in real-world edge inference scenarios?
• RQ5 Does imposing PBP structure during training cause significant accuracy degradation on standard vision benchmarks like MNIST and CIFAR-10?

Key findings

• The PBP-based approach achieves near-linear speedup with coefficient reduction on both NVIDIA Jetson TX1 and ARM Mali T880, with speedups up to 14.09× over cuBLAS and 9.26× over cuSPARSE at 4096×4096 matrix size and 3.125% fill-in.
• On the Jetson TX1, the PBP gemv kernel achieves 100% load and store efficiency, matching dense matrix multiplication, while standard CSR format achieves only 12.5% store efficiency.
• For fill-in factors of 6.25%–12.5% and matrix sizes ≥128×128, permutation overhead is negligible (≤50% of kernel time), making the method practical for typical FC layer sparsity.
• On MNIST, PBP pruning maintains 100% of the original network accuracy; on CIFAR-10, it stays within 0.5% accuracy drop, demonstrating minimal performance loss.
• The PBP form enables significant performance gains over vendor libraries, especially in the 6.25%–25% fill-in range, where generic sparse libraries underperform due to poor memory access patterns.
• Resource utilization measurements show that PBP kernels achieve 100% load and store efficiency on GPU, outperforming CSR (12.5% store efficiency) and BRC (12.52% load efficiency) formats