[Paper Review] How Can We Be So Dense? The Benefits of Using Highly Sparse Representations

Subutai Ahmad, Luiz Scheinkman|arXiv (Cornell University)|Mar 27, 2019

Generative Adversarial Networks and Image Synthesis29 references64 citations

TL;DR

The paper argues for high-dimensional sparse representations and introduces sparse deep networks with k-winners and boosting, showing improved noise robustness on MNIST and Google Speech Commands with competitive accuracy.

ABSTRACT

Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and activations. Simulations on MNIST and the Google Speech Command Dataset show that such networks demonstrate significantly improved robustness and stability compared to dense networks, while maintaining competitive accuracy. We discuss the potential benefits of sparsity on accuracy, noise robustness, hyperparameter tuning, learning speed, computational efficiency, and power requirements.

Motivation & Objective

• Motivate the shift from dense to sparse representations by analyzing robustness to noise and interference.
• Propose a scalable sparse network architecture with sparse weights and activations to exploit high dimensionality.
• Demonstrate robustness improvements on MNIST and Google Speech Commands while preserving accuracy.
• Discuss potential benefits for learning speed, power efficiency, and hyperparameter tuning.

Proposed method

• Develop theoretical properties of high-dimensional sparse representations and matching volumes.
• Define a sparse network with sparse weight matrices and a k-winners activation scheme.
• Incorporate a boosting mechanism to equalize unit usage and maximize representational entropy.
• Apply the sparse architecture to CNNs and fully connected networks in MNIST and Google Speech Commands setups.
• Train using standard back-propagation with tuned hyperparameters and compare against dense baselines.
• Provide discussion on computational efficiency and potential hardware advantages.

Experimental results

Research questions

• RQ1 Do high-dimensional sparse representations improve robustness to noise and interference compared to dense representations?
• RQ2 Can a practically realizable sparse network (with sparse weights and k-winners activations) maintain accuracy while enhancing robustness?
• RQ3 How do sparse networks perform on image and audio benchmarks (MNIST and Google Speech Commands) under noisy conditions compared to dense networks?
• RQ4 What are the practical computational and hardware considerations when implementing sparse networks in standard deep learning frameworks?

Key findings

• Sparse representations exhibit robustness to noise and interference as dimensionality increases when sparsity is maintained.
• A sparse deep network with random sparse weights and k-winners activations achieves competitive accuracy while significantly improving noise robustness on MNIST and Google Speech Commands.
• Introducing sparsity at multiple levels (weights and activations) yields substantial reductions in non-zero multiplies and potential power savings, with robustness benefits.
• Dropout is not consistently beneficial for sparse networks and can diminish robustness in some settings; sparsity itself contributes to robustness.
• The results generalize from MNIST to a speech command dataset, indicating broader applicability of sparse representations across domains.