[Paper Review] Pseudoinverse-free randomized extended block Kaczmarz for solving least squares.

Kui Du, Wutao Si+1 more|arXiv (Cornell University)|Jan 13, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes a pseudoinverse-free randomized extended block Kaczmarz algorithm for solving least squares problems, achieving exponential convergence in mean square to the minimum ℓ₂-norm solution. It works by iteratively projecting block rows of the system matrix onto the solution space without requiring matrix inversion, enabling efficient distributed computation across all system types—consistent or inconsistent, overdetermined or underdetermined, full-rank or rank-deficient.

ABSTRACT

Randomized iterative algorithms have recently been proposed to solve large-scale linear systems. In this paper, we present a simple randomized extended block Kaczmarz algorithm that exponentially converges in the mean square to the unique minimum $\ell_2$-norm least squares solution of a given linear system of equations. The proposed algorithm is pseudoinverse-free and therefore different from the projection-based randomized double block Kaczmarz algorithm of Needell, Zhao, and Zouzias. We emphasize that our method works for all types of linear systems (consistent or inconsistent, overdetermined or underdetermined, full-rank or rank-deficient). Moreover, our approach can utilize efficient implementations on distributed computing units, yielding remarkable improvements in computational time. Numerical examples are given to show the efficiency of the new algorithm.

Motivation & Objective

• To develop a randomized iterative method for solving large-scale least squares problems without relying on matrix pseudoinverses.
• To ensure exponential convergence in mean square to the minimum ℓ₂-norm solution across all linear system configurations.
• To enable efficient implementation on distributed computing architectures for improved computational performance.
• To provide a unified framework applicable to consistent, inconsistent, overdetermined, underdetermined, full-rank, and rank-deficient systems.

Proposed method

• The algorithm uses block row sampling from the system matrix to iteratively refine the solution estimate.
• It applies a randomized Kaczmarz-type update that avoids explicit computation of the Moore-Penrose pseudoinverse.
• The update rule is derived from the extended Kaczmarz framework, projecting onto the hyperplanes defined by selected blocks.
• The method maintains numerical stability and convergence by leveraging block structure and randomization in selection.
• The algorithm is designed to be easily parallelized and distributed across multiple computing units.
• Convergence is analyzed in expectation, showing exponential mean square convergence to the minimum ℓ₂-norm solution.

Experimental results

Research questions

• RQ1 Can a randomized block Kaczmarz method achieve exponential convergence without using pseudoinverse computations?
• RQ2 How does the algorithm perform across diverse linear system types, including inconsistent and rank-deficient systems?
• RQ3 To what extent can the algorithm be scaled efficiently on distributed computing platforms?
• RQ4 What is the theoretical convergence behavior of the method in terms of mean square error decay?
• RQ5 How does the proposed method compare in efficiency and robustness to existing projection-based double block Kaczmarz approaches?

Key findings

• The algorithm achieves exponential convergence in mean square to the minimum ℓ₂-norm least squares solution.
• It is applicable to all types of linear systems, including inconsistent, overdetermined, underdetermined, full-rank, and rank-deficient systems.
• The method is pseudoinverse-free, avoiding costly and numerically unstable matrix inversion operations.
• The algorithm enables significant speedups through efficient distributed computing implementations.
• Numerical experiments confirm the algorithm's efficiency and robustness across various test cases.