[Paper Review] WSNet: Compact and Efficient Networks Through Weight Sampling

Xiaojie Jin, Shuicheng Yan+5 more|arXiv (Cornell University)|Nov 28, 2017

Music and Audio Processing41 references3 citations

TL;DR

WSNet introduces a novel weight sampling architecture that learns compact and efficient deep neural networks by sampling filters from a small set of condensed parameters, enabling significant model compression and computation reduction. The method achieves up to 180× smaller model size and up to 16× faster inference with negligible accuracy drop on audio classification tasks, outperforming state-of-the-art baselines.

ABSTRACT

We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces {parameter sharing} throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180 times smaller and theoretically up to 16 times faster than the well-established baselines, without noticeable performance drop.

Motivation & Objective

• To address the high model size and computational cost of deep neural networks in resource-constrained environments.
• To overcome limitations of existing model compression techniques that treat weights independently and ignore structural correlations.
• To develop a unified training strategy that enforces parameter sharing from the outset, rather than applying compression post-training.
• To enable efficient learning of small, high-performance 1D and 2D convolutional networks for audio and vision tasks.
• To demonstrate the effectiveness of weight sampling with integral image-based computation acceleration for both training and inference.

Proposed method

• WSNet parameterizes convolutional layers using a set of condensed filters that are significantly smaller than standard filters.
• Each actual filter in the network is sampled from the condensed filters via spatial and channel-wise sampling, enforcing parameter sharing from the beginning of training.
• The method uses overlapping sampled filters to exploit redundancy and enable efficient computation using an integral image-based acceleration technique.
• The integral image method reduces computation cost by efficiently computing overlapping filter responses, minimizing redundant operations.
• Weight quantization is applied in conjunction with weight sampling to further compress model size without significant accuracy loss.
• The architecture is extended to 2D CNNs using the same sampling and integral image principles, demonstrating generalization beyond 1D tasks.

Experimental results

Research questions

• RQ1 Can a deep learning architecture be designed from the ground up to enforce parameter sharing during training, rather than applying it post-hoc?
• RQ2 Can weight sampling from a compact set of filters achieve superior model compression and speedup compared to existing methods like pruning, quantization, or factorization?
• RQ3 Does the use of overlapping sampled filters enable both high efficiency and strong learning capacity in compact networks?
• RQ4 Can the integral image method effectively accelerate both training and inference in such a sampled filter architecture?
• RQ5 How does WSNet compare to state-of-the-art models in terms of accuracy, model size, and inference speed on audio and vision benchmarks?

Key findings

• On the ESC-50 dataset, WSNet achieved 66.5% accuracy with a model size of only 0.52M parameters, representing a 100× reduction compared to the baseline.
• With additional weight quantization, WSNet reduced model size to 0.07M parameters (180× smaller than baseline) while maintaining 65.8% accuracy.
• On UrbanSound8K and DCASE datasets, WSNet achieved comparable or higher accuracy than baselines while reducing model size by up to 100×.
• On CIFAR-10 and MNIST, WSNet achieved lower error rates than HashNet at the same compression ratio, demonstrating superior performance under equivalent compression.
• When applied to ResNet50, WSNet achieved a 9× smaller model size (0.85M to 0.094M) with only a 0.5% drop in top-1 accuracy.
• The integral image-based computation method enabled up to 16× faster inference by reducing redundant computations in overlapping filter operations.