[Paper Review] Sketching and Neural Networks

Amit Daniely, Nevena Lazic+2 more|arXiv (Cornell University)|Apr 19, 2016

Machine Learning and Algorithms24 references3 citations

TL;DR

This paper proposes a compact linear sketching method combined with a single-layer neural network to efficiently learn sparse polynomial functions on high-dimensional binary data. By using improper learning with a small sketch size of $O(k\log(d/\delta))$, it achieves provably accurate classification of $1-\delta$ fraction of $k$-sparse inputs, outperforming prior methods that required exponential sketch sizes with respect to polynomial degree.

ABSTRACT

High-dimensional sparse data present computational and statistical challenges for supervised learning. We propose compact linear sketches for reducing the dimensionality of the input, followed by a single layer neural network. We show that any sparse polynomial function can be computed, on nearly all sparse binary vectors, by a single layer neural network that takes a compact sketch of the vector as input. Consequently, when a set of sparse binary vectors is approximately separable using a sparse polynomial, there exists a single-layer neural network that takes a short sketch as input and correctly classifies nearly all the points. Previous work has proposed using sketches to reduce dimensionality while preserving the hypothesis class. However, the sketch size has an exponential dependence on the degree in the case of polynomial classifiers. In stark contrast, our approach of using improper learning, using a larger hypothesis class allows the sketch size to have a logarithmic dependence on the degree. Even in the linear case, our approach allows us to improve on the pesky $O({1}/{γ^2})$ dependence of random projections, on the margin $γ$. We empirically show that our approach leads to more compact neural networks than related methods such as feature hashing at equal or better performance.

Motivation & Objective

• To address the computational and statistical challenges of high-dimensional sparse data in supervised learning.
• To reduce model size and training cost while preserving classification accuracy for sparse polynomial functions.
• To overcome the exponential sketch size dependence of prior methods on polynomial degree by using improper learning with neural networks.
• To design a sparsity-preserving sketching scheme that enables efficient, low-dimensional representation of sparse inputs.
• To empirically validate that sketching with neural networks yields smaller, faster, and more accurate models than feature hashing or random projections.

Proposed method

• Uses compact linear sketches of size $O(k\log(d/\delta))$ to reduce dimensionality of $k$-sparse binary inputs.
• Employs a single-layer neural network with ReLU activation to decode the sketch and compute any $s$-sparse linear or polynomial function.
• Applies sparse sketching matrices to preserve sparsity and enable efficient computation.
• Uses improper learning to allow a larger hypothesis class, enabling logarithmic dependence on polynomial degree instead of exponential.
• Leverages the Johnson-Lindenstrauss lemma and sketching theory (e.g., Count-Min, Count-Sketch) to ensure inner product preservation.
• Employs $\ell_1$-norm regularization and proximal stochastic gradient for parameter sparsity and faster training.

Experimental results

Research questions

• RQ1 Can a compact sketch of high-dimensional sparse binary data preserve the ability to compute any sparse polynomial function?
• RQ2 Does using a neural network to decode the sketch enable significantly smaller model sizes than traditional sketching or hashing methods?
• RQ3 Can the sketch size be reduced to logarithmic dependence on polynomial degree while maintaining high accuracy?
• RQ4 How does sketching with neural networks compare to feature hashing and random projections in terms of model size and performance?
• RQ5 Does the proposed method maintain generalization performance on real-world datasets with limited labeled data?

Key findings

• The method achieves $1-\delta$ fraction classification accuracy on $k$-sparse binary inputs using a sketch size of $O(k\log(d/\delta))$, significantly smaller than prior methods.
• For degree-$p$ polynomial functions, the sketch size depends logarithmically on $p$, in contrast to the exponential dependence in previous approaches.
• Empirical results show that sketching with neural networks leads to models with fewer non-zero parameters and comparable or better accuracy than feature hashing and Gaussian projections.
• On the entity type tagging task, sketching outperformed a single 500,000-size hash function in both accuracy and parameter efficiency.
• On Reuters and AG News datasets, sketching achieved similar or better performance with up to 50% fewer parameters in the first layer.
• Multiple sketch blocks ($t \geq 2$) consistently outperformed single-block sketches at similar model sizes, indicating benefits of redundancy and diversity.