[Paper Review] Improving the numerical stability of fast matrix multiplication algorithms.

Grey Ballard, Austin R. Benson+3 more|arXiv (Cornell University)|Jul 2, 2015

Tensor decomposition and applications19 references4 citations

TL;DR

This paper improves the numerical stability of practical fast matrix multiplication algorithms by tightening theoretical error bounds, introducing diagonal scaling to mitigate inaccuracies from anomalous inputs, and empirically validating accuracy and performance gains. It demonstrates that the numerical cost of fast algorithms is acceptable and can be further reduced through algorithmic enhancements.

ABSTRACT

Fast algorithms for matrix multiplication, or those that perform asymptotically fewer scalar operations than the classical algorithm, have been considered primarily of theoretical interest. Aside from Strassen's original algorithm, few fast algorithms have been efficiently implemented or used in practical applications. However, there exist many practical alternatives to Strassen's algorithm with varying performance and numerical properties. While fast algorithms are known to be numerically stable, their error bounds are slightly weaker than the classical algorithm. We argue in this paper that the numerical sacrifice of fast algorithms, particularly for the typical use cases of practical algorithms, is not prohibitive, and we explore ways to improve the accuracy both theoretically and empirically. The numerical accuracy of fast matrix multiplication depends on properties of the algorithm and of the input matrices, and we consider both contributions independently. We generalize and tighten previous error analyses of fast algorithms, compare the properties among the class of known practical fast algorithms, and discuss algorithmic techniques for improving the error guarantees. We also present means for reducing the numerical inaccuracies generated by anomalous input matrices using diagonal scaling matrices. Finally, we include empirical results that test the various improvement techniques, in terms of both their numerical accuracy and their performance.

Motivation & Objective

• To address the long-standing perception that fast matrix multiplication algorithms are numerically less stable than classical methods.
• To analyze and improve the numerical accuracy of practical fast algorithms beyond existing theoretical bounds.
• To investigate how input matrix properties affect numerical errors in fast matrix multiplication.
• To develop and evaluate algorithmic techniques—particularly diagonal scaling—that reduce numerical inaccuracies.
• To empirically validate both accuracy improvements and performance trade-offs of the proposed techniques.

Proposed method

• Generalizing and tightening prior error analyses for fast matrix multiplication algorithms to provide more precise theoretical bounds.
• Analyzing the impact of input matrix structure on numerical errors, identifying cases where instability arises.
• Introducing diagonal scaling matrices as a preprocessing technique to reduce numerical inaccuracies caused by ill-conditioned inputs.
• Designing and implementing empirical benchmarks to evaluate numerical accuracy and performance across multiple fast algorithms.
• Comparing the numerical properties of known practical fast algorithms to identify trade-offs in stability and efficiency.
• Using both theoretical analysis and empirical testing to assess the effectiveness of scaling and algorithmic modifications.

Experimental results

Research questions

• RQ1 How do the theoretical error bounds of practical fast matrix multiplication algorithms compare to those of the classical algorithm?
• RQ2 What role do input matrix properties play in amplifying numerical errors in fast matrix multiplication?
• RQ3 Can diagonal scaling effectively reduce numerical inaccuracies in fast matrix multiplication for anomalous input matrices?
• RQ4 What is the trade-off between numerical accuracy and performance when applying error-reduction techniques to fast algorithms?
• RQ5 Which fast matrix multiplication algorithm offers the best balance of accuracy, stability, and performance under practical conditions?

Key findings

• The numerical error bounds of fast matrix multiplication algorithms can be generalized and tightened, showing improved theoretical stability.
• Input matrix structure significantly influences numerical errors, with certain matrices amplifying instability in fast algorithms.
• Diagonal scaling reduces numerical inaccuracies caused by anomalous input matrices, improving error guarantees without sacrificing performance.
• Empirical results confirm that the proposed techniques enhance numerical accuracy while maintaining competitive performance.
• The numerical sacrifice of fast algorithms over classical methods is not prohibitive in typical use cases, especially when enhanced with scaling and refined error analysis.
• Among known practical fast algorithms, those with better-conditioned intermediate computations exhibit stronger numerical stability.