[Paper Review] Detecting Dead Weights and Units in Neural Networks

Utku Evci|arXiv (Cornell University)|Jun 15, 2018

Advanced Neural Network Applications28 references4 citations

TL;DR

This paper proposes a novel method to detect and prune dead units—neurons that no longer contribute to learning—in neural networks using Mean Replacement Score (MRS), an efficient approximation of loss change via bias propagation. The approach achieves up to 5x model size reduction on MNIST with no performance drop, demonstrating that unit-wise pruning outperforms traditional parameter-level pruning.

ABSTRACT

Deep Neural Networks are highly over-parameterized and the size of the neural networks can be reduced significantly after training without any decrease in performance. One can clearly see this phenomenon in a wide range of architectures trained for various problems. Weight/channel pruning, distillation, quantization, matrix factorization are some of the main methods one can use to remove the redundancy to come up with smaller and faster models. This work starts with a short informative chapter, where we motivate the pruning idea and provide the necessary notation. In the second chapter, we compare various saliency scores in the context of parameter pruning. Using the insights obtained from this comparison and stating the problems it brings we motivate why pruning units instead of the individual parameters might be a better idea. We propose some set of definitions to quantify and analyze units that don't learn and create any useful information. We propose an efficient way for detecting dead units and use it to select which units to prune. We get 5x model size reduction through unit-wise pruning on MNIST.

Motivation & Objective

• To identify and remove dead units—neurons that stop learning and contribute no useful information—during or after training.
• To improve model efficiency by pruning entire units rather than individual parameters, leading to more compact and faster models.
• To develop an efficient, second-order-inspired method for detecting dead units without expensive Hessian computation.
• To demonstrate that unit pruning yields better compression than parameter-level pruning while preserving model accuracy.
• To provide a practical, open-source PyTorch-based library for unit and parameter pruning with MRS-based saliency scoring.

Proposed method

• Introduces Mean Replacement Score (MRS), an efficient first-order approximation of the loss change due to replacing a unit's output with its mean.
• Uses bias propagation to estimate the impact of removing a unit on the overall loss, enabling fast and accurate saliency scoring.
• Employs a masking mechanism to simulate pruning without retraining, allowing efficient evaluation of unit removal effects.
• Applies MRS to rank units by their contribution to loss, selecting the least important ones for removal.
• Implements a two-stage pruning strategy: first detect dead units via MRS, then remove them using a differentiable masking layer.
• Develops a PyTorch library, `pytorchpruner`, supporting both parameter-level and unit-level pruning with MRS and Hessian-based scoring.

Experimental results

Research questions

• RQ1 Can dead units be reliably detected during or after training using an efficient, differentiable approximation of loss change?
• RQ2 Does pruning entire units instead of individual parameters lead to better model compression with no performance degradation?
• RQ3 How does MRS compare to traditional saliency measures (e.g., L2 norm, gradient magnitude) in identifying non-contributing units?
• RQ4 What is the impact of high learning rates on unit death, and can MRS detect such units early?
• RQ5 Can MRS-based unit pruning achieve significant model compression on standard vision benchmarks like MNIST and CIFAR-10?

Key findings

• MRS-based unit pruning achieved a 5x reduction in model size on the MNIST dataset with no drop in test accuracy.
• Units that die due to high learning rates or activation saturation (e.g., ReLU, Tanh) can be reliably detected using MRS.
• MRS outperformed norm-based saliency measures in identifying non-contributing units, especially in deeper or more complex networks.
• The method detected dead units even in networks with BatchNorm and ReLU, where traditional gradient-based methods fail.
• The proposed `pytorchpruner` library enables efficient, reproducible pruning with support for both parameter and unit-level strategies.
• Pruning via MRS was stable and effective across different architectures and activation functions, including ReLU and Tanh.