[Paper Review] Compressing Neural Networks with the Hashing Trick

Wenlin Chen, James T. Wilson+3 more|arXiv (Cornell University)|Apr 19, 2015

Neural Networks and Applications53 references565 citations

TL;DR

This paper introduces HashedNets, a neural network compression technique that uses a hash function to group connections into shared weight buckets, drastically reducing model size with minimal accuracy loss. By allowing 'virtual' connections through parameter sharing, HashedNets achieve up to 16× model inflation with only 1.61% test error on MNIST, outperforming low-rank and edge-removal baselines at high compression rates.

ABSTRACT

As deep nets are increasingly used in applications suited for mobile devices, a fundamental dilemma becomes apparent: the trend in deep learning is to grow models to absorb ever-increasing data set sizes; however mobile devices are designed with very little memory and cannot store such large models. We present a novel network architecture, HashedNets, that exploits inherent redundancy in neural networks to achieve drastic reductions in model sizes. HashedNets uses a low-cost hash function to randomly group connection weights into hash buckets, and all connections within the same hash bucket share a single parameter value. These parameters are tuned to adjust to the HashedNets weight sharing architecture with standard backprop during training. Our hashing procedure introduces no additional memory overhead, and we demonstrate on several benchmark data sets that HashedNets shrink the storage requirements of neural networks substantially while mostly preserving generalization performance.

Motivation & Objective

• Address the growing memory bottleneck in deploying large deep neural networks on mobile and embedded devices with limited RAM.
• Exploit inherent redundancy in neural network weights to reduce model size without significant performance degradation.
• Develop a memory-efficient compression method that imposes no additional memory overhead during training or inference.
• Demonstrate that parameter sharing via hashing can improve generalization when combined with network inflation.
• Enable deployment of large models on resource-constrained devices by enabling high compression with minimal accuracy drop.

Proposed method

• Apply a random hash function to map neural network connections into shared weight buckets, where all connections in the same bucket use the same parameter value.
• Use standard backpropagation to train the shared parameters, allowing gradients to update all connections in a bucket simultaneously.
• Introduce a network inflation mechanism by increasing the number of virtual hidden nodes while keeping the number of real parameters fixed.
• Maintain compatibility with standard deep learning techniques such as dropout, ReLU activation, and weight sparsity.
• Use a fixed number of real parameters (K^ℓ) while varying the number of virtual nodes (n^ℓ) to study the effect of expansion on generalization.
• Optimize the hashing process to minimize collisions and maintain gradient signal diversity during training.

Experimental results

Research questions

• RQ1 Can parameter sharing via hashing significantly reduce neural network model size with minimal impact on test accuracy?
• RQ2 How does network inflation through virtual connections affect generalization performance under fixed memory constraints?
• RQ3 Does hashing-based compression outperform low-rank decomposition and edge removal in high-compression regimes?
• RQ4 What is the optimal expansion factor (in terms of virtual nodes) that maximizes performance without increasing real parameter count?
• RQ5 How does the method scale across different benchmark datasets and network architectures?

Key findings

• HashedNets achieve up to 16× virtual network inflation with only a 1.61% test error on MNIST, representing a 50% relative reduction in error compared to a standard network.
• At a compression factor of 1/64, HashedNets outperform low-rank decomposition and edge removal on all eight benchmark datasets, especially in extreme compression scenarios.
• The method maintains high performance even when only 0.5 bits of information are stored per virtual parameter, demonstrating extreme compression efficiency.
• Best performance is achieved with an inflation factor of 8× to 16×, after which further expansion leads to increased test error due to excessive hash collisions.
• HashedNets show superior robustness to compression compared to low-rank and edge-removal baselines, particularly at very high compression rates.
• The approach imposes no additional memory overhead and is compatible with standard regularization techniques like dropout and weight sparsity.