[Paper Review] What if Neural Networks had SVDs?

Alexander Mathiasen, Frederik Hvilshøj+3 more|arXiv (Cornell University)|Sep 29, 2020

Neural Networks and Applications13 references4 citations

TL;DR

This paper proposes FastH, a novel parallel algorithm that accelerates matrix operations in neural networks by efficiently computing orthogonal matrix multiplications via Householder decompositions. By increasing parallelism in GPU execution, FastH achieves up to 27× speedup over prior sequential methods while preserving the same theoretical time complexity and output as existing SVD reparameterization techniques.

ABSTRACT

Various Neural Networks employ time-consuming matrix operations like matrix inversion. Many such matrix operations are faster to compute given the Singular Value Decomposition (SVD). Previous work allows using the SVD in Neural Networks without computing it. In theory, the techniques can speed up matrix operations, however, in practice, they are not fast enough. We present an algorithm that is fast enough to speed up several matrix operations. The algorithm increases the degree of parallelism of an underlying matrix multiplication $H\cdot X$ where $H$ is an orthogonal matrix represented by a product of Householder matrices. Code is available at www.github.com/AlexanderMath/fasth .

Motivation & Objective

• To address the performance bottleneck in SVD-based matrix operations within neural networks, particularly on GPUs.
• To overcome the inefficiency of sequential inner product computation in prior Householder-based SVD reparameterization methods.
• To design a parallel algorithm that maintains the same mathematical output as existing methods but significantly improves runtime on GPU hardware.
• To enable practical speedups for matrix inversion, determinant computation, and other SVD-accelerated operations in deep learning.
• To provide a drop-in replacement for existing SVD reparameterization frameworks with minimal code changes.

Proposed method

• FastH restructures the computation of orthogonal matrix multiplication $ H \cdot X $, where $ H $ is a product of $ d $ Householder matrices, to increase parallelism.
• Instead of sequentially applying $ d $ Householder transformations, FastH reorganizes the computation to reduce sequential dependencies and better utilize GPU cores.
• The algorithm performs $ O(d/m + m) $ sequential matrix-matrix operations for a mini-batch of size $ m $, reducing the number of sequential inner products from $ O(d) $.
• FastH is implemented in CUDA to fully exploit GPU parallelism, avoiding the performance limitations of Python-based implementations.
• The method retains the same mathematical output as previous SVD reparameterization techniques, ensuring compatibility with existing models.
• A PyTorch-compatible implementation is provided, allowing users to replace `nn.Linear` with `LinearSVD` with minimal code changes.

Experimental results

Research questions

• RQ1 Can a parallel algorithm be designed to accelerate Householder-based matrix multiplication in neural networks without increasing time complexity?
• RQ2 Why do prior SVD reparameterization methods fail to deliver practical speedups on GPUs despite favorable theoretical time complexity?
• RQ3 To what extent can increased parallelism reduce sequential bottlenecks in GPU-accelerated matrix operations involving orthogonal matrices?
• RQ4 Can the same mathematical output as existing SVD reparameterization be achieved with significantly improved runtime on modern hardware?
• RQ5 Is it possible to achieve practical speedups for SVD-accelerated operations like matrix inversion and determinant computation in deep learning?

Key findings

• FastH achieves up to 27× speedup over the sequential algorithm from [17] in matrix inversion benchmarks on GPU hardware.
• The algorithm reduces the number of sequential operations from $ O(d) $ inner products to $ O(d/m + m) $ matrix-matrix operations, significantly improving GPU core utilization.
• FastH maintains the same output and theoretical time complexity as prior methods, ensuring correctness and compatibility with existing SVD reparameterization frameworks.
• The implementation in CUDA enables full GPU parallelism, overcoming the performance limitations of CPU-based or sequential approaches.
• The method is compatible with existing deep learning models, requiring only a single-line code change to integrate into PyTorch workflows.
• By reducing computation time, FastH lowers energy consumption and carbon emissions, especially benefiting researchers with limited computational resources.