[Paper Review] Structured Transforms for Small-Footprint Deep Learning

Vikas Sindhwani, Tara N. Sainath+1 more|arXiv (Cornell University)|Oct 6, 2015

Speech and Audio Processing26 references102 citations

TL;DR

This paper proposes a unified framework for learning structured parameter matrices with low displacement rank to enable efficient, compact deep learning for mobile deployment. By leveraging structured transforms like Toeplitz, Vandermonde, and Cauchy matrices, the method accelerates inference and training while achieving superior accuracy-compactness-speed trade-offs, reducing model size by over 3.5× with near-state-of-the-art performance in keyword spotting.

ABSTRACT

We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices. We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank. Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured. Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques. In keyword spotting applications in mobile speech recognition, our methods are much more effective than standard linear low-rank bottleneck layers and nearly retain the performance of state of the art models, while providing more than 3.5-fold compression.

Motivation & Objective

• To address the challenge of deploying large deep learning models on power- and storage-constrained mobile and wearable devices.
• To develop a unified framework for learning structured parameter matrices that reduce model size and accelerate computation.
• To enable explicit tuning of statistical modeling capacity from structured to unstructured via displacement rank.
• To outperform existing techniques like low-rank bottlenecks, Circulant, and Fastfood transforms in accuracy, compression, and inference speed.

Proposed method

• The framework learns parameter matrices with low displacement rank, a property that enables fast matrix-vector multiplication and gradient computation.
• It generalizes multiple structured matrix classes—Toeplitz, Vandermonde, Cauchy—by modeling them under a common algebraic framework.
• The method supports efficient function and gradient evaluation using fast algorithms specific to each matrix class, reducing O(mn) operations to O(r(m+n)) for rank-r approximations.
• Parameter sharing is explicitly controlled via displacement rank, allowing a continuum from highly structured to nearly unstructured matrices.
• The approach is integrated into deep learning pipelines, particularly in fully connected layers, to replace dense matrices with compact, structured alternatives.
• Training uses standard stochastic optimization with layer-specific learning rates, enabling end-to-end learning of structured transforms.

Experimental results

Research questions

• RQ1 Can structured transforms with low displacement rank achieve better accuracy-compactness-speed trade-offs than standard low-rank bottlenecks in mobile deep learning?
• RQ2 To what extent can structured matrices like Toeplitz, Vandermonde, and Cauchy reduce model size while maintaining performance?
• RQ3 How does displacement rank control the modeling capacity and generalization ability of structured transforms?
• RQ4 Can structured transforms outperform specialized structured matrices like Circulant and Fastfood in real-world mobile applications?

Key findings

• A displacement rank=1 Toeplitz-like structured transform with only 3,348 parameters outperforms a rank=16 low-rank bottleneck (with 53,568 parameters) in keyword spotting, reducing false reject rate from 10.2% to 8.2%.
• With displacement rank=10, the model achieves a false reject rate of 6.2%, outperforming a larger rank=32 low-rank model (6.8%)
• The best structured transform model comes within 0.4% of the performance of a fully-connected model that is 80× larger and a reference model that is 3.6× larger.
• The method achieves classification accuracy within 0.2% of the fully-connected model and reference model, while training significantly faster than standard low-rank baselines.
• In noisy conditions (e.g., cafe0 dataset with 0dB babble noise), the performance gains of structured transforms are consistent and robust.