[Paper Review] Structured adaptive and random spinners for fast machine learning computations

Mariusz Bojarski, Anna Choromanska+7 more|arXiv (Cornell University)|Oct 19, 2016

Stochastic Gradient Optimization Techniques21 citations

TL;DR

This paper introduces Structured Spinners—efficient, parameterized matrices formed as products of three structured blocks incorporating rotations—to accelerate machine learning computations. By replacing dense projection matrices with these structured alternatives, the method achieves sub-quadratic space complexity and O(n log m) computation time via Fast Fourier Transform techniques, enabling significant speedups in kernel approximation, dimensionality reduction, LSH, deep learning, and convex optimization with theoretical guarantees and empirical validation across diverse applications.

ABSTRACT

We consider an efficient computational framework for speeding up several machine learning algorithms with almost no loss of accuracy. The proposed framework relies on projections via structured matrices that we call Structured Spinners, which are formed as products of three structured matrix-blocks that incorporate rotations. The approach is highly generic, i.e. i) structured matrices under consideration can either be fully-randomized or learned, ii) our structured family contains as special cases all previously considered structured schemes, iii) the setting extends to the non-linear case where the projections are followed by non-linear functions, and iv) the method finds numerous applications including kernel approximations via random feature maps, dimensionality reduction algorithms, new fast cross-polytope LSH techniques, deep learning, convex optimization algorithms via Newton sketches, quantization with random projection trees, and more. The proposed framework comes with theoretical guarantees characterizing the capacity of the structured model in reference to its unstructured counterpart and is based on a general theoretical principle that we describe in the paper. As a consequence of our theoretical analysis, we provide the first theoretical guarantees for one of the most efficient existing LSH algorithms based on the HD3HD2HD1 structured matrix [Andoni et al., 2015]. The exhaustive experimental evaluation confirms the accuracy and efficiency of structured spinners for a variety of different applications.

Motivation & Objective

• To address the high computational and memory costs of dense projection matrices in machine learning algorithms.
• To develop a generic, structured framework that maintains accuracy while drastically reducing time and space complexity.
• To provide theoretical justification for the effectiveness of structured matrices, including previously unexplained methods like HD3HD2HD1 in cross-polytope LSH.
• To unify and generalize existing structured matrix approaches under a single theoretical principle.
• To validate the framework across multiple ML applications, including random feature maps, Newton sketches, and quantization.

Proposed method

• Proposes Structured Spinners as products of three matrix blocks, each incorporating rotations, to form highly parameterized structured matrices.
• Uses Hadamard matrices (H) combined with random or adaptive diagonal matrices (D) to form matrices like HD3HD2HD1, enabling fast computation via FFT.
• Applies the framework to various ML tasks: kernel approximation via random feature maps, dimensionality reduction, cross-polytope LSH, deep learning, and Newton sketches.
• Employs theoretical analysis based on a general principle to characterize the capacity of structured models relative to unstructured ones.
• Introduces a novel theoretical framework that explains the success of structured neural networks and provides first guarantees for HD3HD2HD1-based LSH.
• Employs wall-clock time and convergence metrics to compare structured spinners against exact and unstructured baselines in optimization and kernel approximation.

Experimental results

Research questions

• RQ1 Can structured matrices be designed to maintain high accuracy while drastically reducing computational and memory costs in machine learning?
• RQ2 Why do structured matrices like HD3HD2HD1 achieve state-of-the-art performance in cross-polytope LSH, and what is their theoretical basis?
• RQ3 Can a unified theoretical framework explain the effectiveness of diverse structured matrix families, including learned and randomized variants?
• RQ4 How do structured spinners compare to exact or unstructured projections in terms of convergence speed and accuracy across optimization and kernel approximation tasks?
• RQ5 What is the theoretical relationship between the capacity of structured models and their unstructured counterparts?

Key findings

• Structured spinners achieve O(n log m) computation time for projections, significantly faster than the O(mn) time of dense matrices, especially when FFT is applied.
• The method reduces space complexity to sub-quadratic, often linear or constant, enabling deployment on memory-constrained systems.
• In Newton sketch optimization, structured spinners reduce complexity from O(nd²) to O(dn log n + md²), with wall-clock times showing substantial gains at lower dimensions.
• The HD3HD2HD1 matrix, used in the fastest known cross-polytope LSH, is theoretically justified for the first time in this work.
• Empirical results show that structured spinners achieve convergence rates comparable to exact methods in logistic regression and kernel approximation, with significant speedups.
• The framework unifies and generalizes prior structured matrix approaches, including the P-model, and provides first theoretical guarantees for structured neural network architectures.