[Paper Review] Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks

Ángel López García-Arias, Masanori Hashimoto+2 more|arXiv (Cornell University)|Nov 24, 2021

Advanced Neural Network Applications33 references4 citations

TL;DR

This paper proposes Hidden-Fold Networks (HFNs), a method that compresses ResNets into highly efficient recurrent architectures by folding residual connections and discovering a high-accuracy subnetwork via sparse supermasks without weight updates. HFN achieves ResNet-level performance on CIFAR100 and ImageNet with 26.8x to 38.5x smaller memory footprint, enabling ultra-low-energy inference on specialized hardware by minimizing off-chip memory access through random, reusable weights and binary supermasks.

ABSTRACT

Deep neural networks (DNNs) are so over-parametrized that recent research has found them to already contain a subnetwork with high accuracy at their randomly initialized state. Finding these subnetworks is a viable alternative training method to weight learning. In parallel, another line of work has hypothesized that deep residual networks (ResNets) are trying to approximate the behaviour of shallow recurrent neural networks (RNNs) and has proposed a way for compressing them into recurrent models. This paper proposes blending these lines of research into a highly compressed yet accurate model: Hidden-Fold Networks (HFNs). By first folding ResNet into a recurrent structure and then searching for an accurate subnetwork hidden within the randomly initialized model, a high-performing yet tiny HFN is obtained without ever updating the weights. As a result, HFN achieves equivalent performance to ResNet50 on CIFAR100 while occupying 38.5x less memory, and similar performance to ResNet34 on ImageNet with a memory size 26.8x smaller. The HFN will become even more attractive by minimizing data transfers while staying accurate when it runs on highly-quantized and randomly-weighted DNN inference accelerators. Code available at https://github.com/Lopez-Angel/hidden-fold-networks

Motivation & Objective

• To address the high memory and energy cost of large DNNs by compressing ResNets into highly efficient, low-memory architectures.
• To leverage the Lottery Ticket Hypothesis and subnetwork discovery in randomly initialized networks to find accurate, untrained subnetworks in folded recurrent structures.
• To enable energy-efficient DNN inference by minimizing off-chip memory access through on-chip storage of sparse supermasks and random seeds.
• To explore whether folding ResNets into recurrent structures enhances the accuracy of hidden subnetworks compared to standard feed-forward models.
• To design a model that is both compact and accurate, suitable for deployment on low-power, highly-quantized inference accelerators.

Proposed method

• Folding a standard ResNet into a recurrent architecture by reusing the same set of weights across multiple time steps, reducing parameter count and memory footprint.
• Using a learnable binary supermask to identify a sparse, high-performing subnetwork within the randomly initialized folded network, without updating the network weights.
• Training the supermask via a differentiable relaxation of the binary mask, optimizing for accuracy while maintaining sparsity.
• Employing a sparse supermask to selectively activate only a subset of the recurrently reused weights during inference, enabling efficient binary operations.
• Utilizing a random seed to generate fixed, non-learned weights, eliminating the need to store parameters in off-chip memory.
• Designing the architecture to be compatible with specialized inference accelerators that support on-chip random number generation and binary mask operations.

Experimental results

Research questions

• RQ1 Can folding a ResNet into a recurrent architecture improve the accuracy of a hidden subnetwork discovered via supermasking compared to standard feed-forward models?
• RQ2 To what extent can the memory footprint of a deep network be reduced by combining weight folding with supermask-based subnetwork discovery?
• RQ3 How does the performance of a hidden-fold network compare to standard ResNets in terms of accuracy and model size on standard benchmarks like CIFAR100 and ImageNet?
• RQ4 Can HFNs achieve significant energy savings by minimizing off-chip memory access through on-chip storage of supermasks and random seeds?
• RQ5 What is the optimal supermask density and training schedule for maximizing HFN accuracy while minimizing model size and computational cost?

Key findings

• HFN achieves comparable accuracy to ResNet50 on CIFAR100 while using only 1/38.5 of the memory, demonstrating a 38.5x reduction in model size.
• On ImageNet, HFN matches the performance of ResNet34 with a memory footprint 26.8x smaller than the original model.
• The energy cost of loading the model from DRAM is reduced by two orders of magnitude on a 45 nm CMOS inference accelerator, due to minimal off-chip memory access.
• Training the supermask for HFN takes 149 seconds per epoch on a single RTX 3090, compared to 44 seconds for the full ResNet50, indicating a trade-off in training cost for massive inference savings.
• The method enables deployment on specialized hardware where parameters are stored in on-chip SRAM using only a random seed and a sparse binary supermask, eliminating the need for DRAM access.
• HFN’s performance is maintained even when using highly quantized weights and binary supermasks, making it ideal for energy-efficient, low-precision inference accelerators.