[Paper Review] $S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks

Xinlin Li, Bang Liu+4 more|arXiv (Cornell University)|Jul 7, 2021

Brain Tumor Detection and Classification4 citations

TL;DR

This paper proposes S³ (Sign-Sparse-Shift) reparameterization, a novel method that decomposes low-bit discrete weights into sign, sparsity, and shift components to mitigate gradient vanishing and weight sign freezing in shift networks. By optimizing in a 3D augmented parameter space, S³ enables 3-bit shift networks to achieve top-1 accuracy on ImageNet that matches full-precision models, eliminating reliance on pre-trained weights and improving training stability and performance.

ABSTRACT

Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy efficient compared to conventional neural networks. However, existing shift networks are sensitive to the weight initialization, and also yield a degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S low-bit re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in a sign-sparse-shift 3-fold manner. In this way, it efficiently learns a low-bit network with a weight dynamics similar to full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks out-performs their full-precision counterparts in terms of top-1 accuracy on ImageNet.

Motivation & Objective

• Address the poor performance and training instability of low-bit shift networks, which suffer from gradient vanishing and weight sign freezing during backpropagation.
• Overcome the sensitivity of shift networks to weight initialization, which currently requires pre-trained full-precision weights for competitive results.
• Develop a reparameterization technique that enables effective training of low-bit shift networks from random initialization without relying on full-precision pretraining.
• Achieve performance parity between 3-bit shift networks and full-precision models on large-scale benchmarks like ImageNet.
• Provide a theoretically grounded, robust training strategy that is insensitive to hyperparameter choices such as learning rate decay and regularization strength.

Proposed method

• Introduce S³ reparameterization, a 3-fold decomposition of discrete weights into sign, sparsity (sparsity parameter), and shift (bit position) components, forming an augmented 3D parameter space.
• Use a dense weight regularizer that maximizes the ℓ₀ norm of the discrete weights during training, promoting non-zero activation and preventing weight sign freezing.
• Apply a straight-through estimator (STE) to approximate gradients through the non-differentiable quantizer, while the S³ decomposition ensures more orthogonal optimization dynamics.
• Regularize the sparsity parameter during training to maintain effective weight dynamics similar to full-precision networks, enhancing convergence and stability.
• Decouple the roles of sign, sparsity, and shift in the optimization process, allowing independent and more effective learning of each component.
• Use a linear or cosine decay scheduler for the dense weight regularizer hyperparameter α, though results show the method is robust to its choice, with most benefit arising in early training.

Experimental results

Research questions

• RQ1 Can low-bit shift networks achieve performance comparable to full-precision models without relying on pre-trained weights?
• RQ2 Does a sign-sparse-shift reparameterization strategy effectively mitigate gradient vanishing and weight sign freezing in low-bit training?
• RQ3 How sensitive is the proposed method to hyperparameters such as the dense weight regularizer strength and learning rate decay schedule?
• RQ4 Can 3-bit shift networks trained with S³ reparameterization match or exceed the top-1 accuracy of full-precision ResNet-18 and ResNet-50 on ImageNet?
• RQ5 Does the S³ method enable stable and convergent training of low-bit shift networks from random initialization, even with frequent sign changes in binary parameters?

Key findings

• S³ reparameterization enables 3-bit shift networks to achieve a top-1 accuracy of 69.83% on ImageNet using ResNet-18, matching the performance of full-precision models.
• With 200 training epochs, the 3-bit S³ network achieves 75.75% top-1 accuracy on ImageNet using ResNet-50, closing the accuracy gap with full-precision counterparts.
• The method is robust to hyperparameter choices: performance remains stable across a wide range of α values (1e-2 to 1e-6) and decay schedulers (linear and cosine).
• Without the dense weight regularizer, training fails to converge, demonstrating that the regularizer is essential for preventing weight sign freezing and enabling stable optimization.
• The performance gain from pre-trained weights in prior methods is largely explained by good initial sign estimation, which S³ achieves inherently through its reparameterization design.
• The method reduces reliance on full-precision pretraining, enabling effective training from random initialization while maintaining high accuracy and robustness.