[Paper Review] Greed Works: An Improved Analysis of Sampling Kaczmarz-Motzkin

Jamie Haddock, Anna Ma|arXiv (Cornell University)|Dec 7, 2019

Stochastic Gradient Optimization Techniques59 references4 citations

TL;DR

This paper presents an improved convergence analysis for the Sampling Kaczmarz-Motzkin (SKM) algorithm, a hybrid randomized-greedy method for solving large-scale linear systems. By removing the need for fixed sample size and normalized rows, the analysis reveals that SKM's convergence rate depends on both the dynamic range of the system and the adaptive choice of sample size β, significantly improving upon prior bounds and demonstrating the benefit of partial greediness in iterative solvers.

ABSTRACT

Stochastic iterative algorithms have gained recent interest in machine learning and signal processing for solving large-scale systems of equations, $Ax=b$. One such example is the Randomized Kaczmarz (RK) algorithm, which acts only on single rows of the matrix $A$ at a time. While RK randomly selects a row of $A$ to work with, Motzkin's Method (MM) employs a greedy row selection. Connections between the two algorithms resulted in the Sampling Kaczmarz-Motzkin (SKM) algorithm which samples a random subset of $β$ rows of $A$ and then greedily selects the best row of the subset. Despite their variable computational costs, all three algorithms have been proven to have the same theoretical upper bound on the convergence rate. In this work, an improved analysis of the range of random (RK) to greedy (MM) methods is presented. This analysis improves upon previous known convergence bounds for SKM, capturing the benefit of partially greedy selection schemes. This work also further generalizes previous known results, removing the theoretical assumptions that $β$ must be fixed at every iteration and that $A$ must have normalized rows.

Motivation & Objective

• To close the gap between purely randomized and fully greedy Kaczmarz methods by improving the theoretical convergence bounds for the Sampling Kaczmarz-Motzkin (SKM) algorithm.
• To generalize prior analyses by removing restrictive assumptions, including fixed β at every iteration and normalized rows in the matrix A.
• To quantify how the convergence rate of SKM depends on the dynamic range of the linear system and the choice of sample size β.
• To demonstrate empirically that intermediate values of β (e.g., β=10) often yield optimal trade-offs between convergence speed and computational cost.
• To open new research directions on adaptive β selection and optimal sample size strategies for SKM-type algorithms.

Proposed method

• Proposes a refined theoretical analysis of SKM that captures the impact of varying sample size βk across iterations, rather than assuming β is fixed.
• Introduces a convergence bound that explicitly depends on the dynamic range of the system, defined as the ratio of the largest to smallest non-zero singular values.
• Removes the assumption that rows of A must be normalized, allowing the analysis to apply to general matrices.
• Uses a probabilistic framework to analyze the expected decrease in error per iteration, leveraging the residual magnitude and row norms.
• Employs a generalized residual selection rule where the algorithm samples β rows and selects the one with the maximum residual magnitude.
• Derives a new upper bound on the expected error decay that improves upon the classical (1 - σ²min / ||A||²F) rate by incorporating β and system structure.

Experimental results

Research questions

• RQ1 How does the convergence rate of the SKM algorithm depend on the sample size β when β is allowed to vary across iterations?
• RQ2 Can the convergence analysis of SKM be improved by removing the assumptions of fixed β and normalized rows?
• RQ3 What is the role of the dynamic range of the linear system in determining the convergence behavior of SKM?
• RQ4 Does partial greediness (intermediate β) lead to better performance than purely random (β=1) or fully greedy (β=m) selection?
• RQ5 What is the optimal choice of β for minimizing computational cost while achieving fast convergence in practice?

Key findings

• The improved convergence bound for SKM depends on both the dynamic range of the system and the adaptive choice of βk, showing that partial greediness enhances convergence speed.
• The analysis removes the restrictive assumptions of fixed β and normalized rows, making the theoretical results applicable to a broader class of linear systems.
• Empirical results show that β=10 achieves optimal performance in terms of CPU time on scale-free network matrices and real-world sparse matrices from SuiteSparse.
• For the 'ash958' matrix, β=10 achieves the lowest CPU time while maintaining high accuracy, indicating that intermediate β values can outperform extreme choices.
• On real-world matrices, increasing β from 1 to 10 to 50 reduces approximation error but increases FLOPS and CPU time, confirming a trade-off between accuracy and efficiency.
• Conjugate Gradient Least Squares (CGLS) outperforms SKM with β=1,10,50 in terms of final error and speed, but SKM is more naturally suited for distributed computing environments.