[Paper Review] Parameter-Efficient Masking Networks

Yue Bai, Huan Wang+4 more|arXiv (Cornell University)|Oct 13, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces Parameter-Efficient Masking Networks (PEMN), a novel network compression paradigm that uses a single random weight vector and learnable masks to represent deep neural networks, significantly reducing model size. By learning diverse masks for fixed random weights with limited unique values, PEMN achieves high performance with low storage cost, outperforming baselines in compression efficiency and robustness across multiple architectures.

ABSTRACT

A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer). They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning diverse masks and introduce the Parameter-Efficient Masking Networks (PEMN). It also naturally leads to a new paradigm for model compression to diminish the model size. Concretely, motivated by the repetitive structures in modern neural networks, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive network modules. Therefore, the model can be expressed as extit{one-layer} with a bunch of masks, which significantly reduce the model storage cost. Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights vector. In this way, our method can further lower the space complexity, especially for models without many repetitive architectures. We validate the potential of PEMN learning masks on random weights with limited unique values and test its effectiveness for a new compression paradigm based on different network architectures. Code is available at https://github.com/yueb17/PEMN

Motivation & Objective

• To investigate the maximum representative potential of fixed random weights with limited unique values in deep neural networks.
• To address the challenge of high model size in deep networks, especially for mobile and transfer-friendly deployment.
• To propose a new network compression paradigm that replaces sparse weight storage with a random vector and learned masks.
• To generalize the compression method across diverse architectures, including those without repetitive structures.
• To demonstrate that model performance remains robust even under high compression ratios using the proposed method.

Proposed method

• Proposes a one-layer strategy using a single randomly initialized layer as a prototype, replicated across the network with different masks.
• Introduces max-layer padding (MP) to fill layers with parameters from the densest layer in the network, preserving structural similarity.
• Develops random vector padding (RP) as an architecture-agnostic method, using a single random vector of appropriate length to initialize all layers.
• Learns task-specific binary or continuous masks via backpropagation to select subnetworks from the fixed random weight structure.
• Represents the full network as a combination of a fixed random weight vector and a set of learned masks, drastically reducing unique parameter count.
• Replaces conventional sparse weight restoration with mask-based reconstruction, lowering storage and transfer costs.

Experimental results

Research questions

• RQ1 What is the maximum representative capacity of a fixed random weight vector with limited unique values when combined with learnable masks?
• RQ2 Can a neural network with minimal unique parameters (via random weight padding) achieve competitive performance through mask learning?
• RQ3 How does the proposed PEMN-based compression paradigm compare to conventional sparse network training in terms of compression ratio and accuracy trade-offs?
• RQ4 To what extent is the proposed method generalizable across different network architectures, including non-repetitive ones?
• RQ5 Can the method maintain performance under extreme compression ratios, especially when compared to larger baseline models?

Key findings

• The proposed PEMN method achieves state-of-the-art compression performance across multiple architectures, including ResNet and ConvMixer, with significantly higher compression ratios than baselines.
• Unlike conventional sparse training, where performance degrades sharply with increasing compression, PEMN remains robust and maintains high accuracy even at high compression levels.
• Compressed small models using PEMN outperform larger baseline models trained with conventional sparsity methods, demonstrating superior parameter efficiency.
• The performance gap between compressed and dense models is smaller in deeper networks (e.g., ResNet56 vs. ResNet32), indicating better scalability with depth.
• Experiments on CIFAR-10 with ConvMixer show that PEMN maintains higher accuracy at lower model sizes compared to baselines, confirming effectiveness across different network designs.
• The method's architecture-agnostic nature via random vector padding enables broad applicability, even to networks without repetitive structures.