[Paper Review] Towards a Theoretical Understanding of Hashing-Based Neural Nets

Yibo Lin, Zhao Song+1 more|arXiv (Cornell University)|Dec 26, 2018

Advanced Neural Network Applications4 citations

TL;DR

This paper provides the first theoretical guarantees for hashing-based neural network compression by showing that random linear sketching via hashing preserves network performance on low-dimensional manifolds. It proves that sketched networks can approximate the original network and that one-hidden-layer HashedNets have a local strong convexity property, enabling provable parameter recovery under mild conditions.

ABSTRACT

Parameter reduction has been an important topic in deep learning due to the ever-increasing size of deep neural network models and the need to train and run them on resource limited machines. Despite many efforts in this area, there were no rigorous theoretical guarantees on why existing neural net compression methods should work. In this paper, we provide provable guarantees on some hashing-based parameter reduction methods in neural nets. First, we introduce a neural net compression scheme based on random linear sketching (which is usually implemented efficiently via hashing), and show that the sketched (smaller) network is able to approximate the original network on all input data coming from any smooth and well-conditioned low-dimensional manifold. The sketched network can also be trained directly via back-propagation. Next, we study the previously proposed HashedNets architecture and show that the optimization landscape of one-hidden-layer HashedNets has a local strong convexity property similar to a normal fully connected neural network. We complement our theoretical results with empirical verifications.

Motivation & Objective

• To bridge the gap between empirical success and theoretical understanding in neural network compression.
• To provide rigorous theoretical justification for hashing-based parameter reduction methods in deep learning.
• To establish approximation guarantees for sketching-based neural network compression using random linear sketching.
• To analyze the optimization landscape of HashedNets and show local strong convexity in one-hidden-layer networks.
• To demonstrate that parameters in HashedNets can be provably recovered under mild assumptions.

Proposed method

• Proposes a neural network compression scheme using random linear sketching, efficiently implemented via hashing functions.
• Uses subspace embedding theory to show that sketching preserves inner products on low-dimensional manifolds with high probability.
• Applies sketching to weight matrices, reducing parameters by enforcing weight sharing across hash buckets.
• Analyzes the optimization landscape of one-hidden-layer HashedNets and proves a local strong convexity property similar to fully connected networks.
• Employs k-wise independent hash functions to ensure uniform bucket distribution and theoretical guarantees.
• Uses concentration inequalities and probabilistic bounds (e.g., Lemma C.12) to analyze hash bucket occupancy and matrix rank properties.

Experimental results

Research questions

• RQ1 Can hashing-based compression preserve the approximation capability of deep neural networks on smooth, low-dimensional data manifolds?
• RQ2 Does the optimization landscape of HashedNets exhibit favorable properties such as local strong convexity?
• RQ3 Can the parameters of a one-hidden-layer HashedNet be provably recovered via gradient descent under mild assumptions?
• RQ4 How does random linear sketching via hashing affect the generalization and training dynamics of compressed networks?
• RQ5 What theoretical guarantees can be provided for the stability and convergence of hashing-based neural networks?

Key findings

• The sketched network approximates the original network on all inputs from any smooth, well-conditioned low-dimensional manifold with high probability.
• The sketching method ensures that inner products between data points are preserved within an ε-approximation, enabling accurate function approximation.
• The one-hidden-layer HashedNet optimization objective exhibits local strong convexity, similar to fully connected networks, under mild conditions.
• A good initialization for HashedNets can be obtained using the algorithm from Zhong et al. (2017b), enabling provable parameter recovery.
• Empirical results confirm that weight matrices in HashedNets maintain favorable numerical properties, such as low condition number and stable rank.
• Theoretical analysis shows that with high probability, hash buckets distribute inputs uniformly when using t-wise independent hashing with t=Θ(log N) or t=Θ(log B).