[Paper Review] On the Relation Between the Randomized Extended Kaczmarz Algorithm and Coordinate Descent

Bogdan Dumitrescu|arXiv (Cornell University)|May 27, 2014

Stochastic Gradient Optimization Techniques6 references4 citations

TL;DR

This paper establishes that randomized coordinate descent (CD) outperforms the randomized extended Kaczmarz (EK) algorithm for solving full-rank overdetermined linear least-squares problems. By showing EK is equivalent to alternating CD and Kaczmarz steps, the authors prove CD converges in fewer iterations and with lower operation count, recommending CD as superior for overdetermined problems and CD followed by Kaczmarz for rank-deficient cases.

ABSTRACT

In this note we compare the randomized extended Kaczmarz (EK) algorithm and randomized coordinate descent (CD) for solving the full-rank overdetermined linear least-squares problem and prove that CD needs less operations for satisfying the same residual-related termination criteria. For the general least-squares problems, we show that running first CD to compute the residual and then standard Kaczmarz on the resulting consistent system is more efficient than EK.

Motivation & Objective

• To analyze the computational relationship between randomized extended Kaczmarz (EK) and coordinate descent (CD) for linear least-squares problems.
• To determine whether EK offers any advantage over CD in terms of convergence speed or computational cost.
• To propose improved algorithmic strategies combining CD and Kaczmarz for better performance on general least-squares problems.
• To provide theoretical and empirical justification for replacing EK with CD or CD+K in practical applications.

Proposed method

• The paper compares EK and CD by decomposing EK into its constituent parts: residual approximation via CD and Kaczmarz iterations on the resulting consistent system.
• It uses the Frobenius norm to define selection probabilities for random column (CD) and row (Kaczmarz) sampling in both algorithms.
• The residual is updated in CD by projecting onto the column space of A, while EK uses a dual residual update mechanism that combines CD and Kaczmarz steps.
• Theoretical analysis shows that CD converges to the least-squares solution independently, while EK relies on CD to build a good residual approximation before applying Kaczmarz.
• Numerical experiments compare normalized RMSE across CD, EK, CD+K, and CD+EK+K over dense and sparse matrices with varying dimensions and ranks.
• The paper introduces a hybrid strategy: first run CD to approximate the residual, then apply standard Kaczmarz to the consistent system, which is shown to be more efficient than EK.

Experimental results

Research questions

• RQ1 Does the randomized extended Kaczmarz algorithm converge faster than coordinate descent in terms of iterations and operations for full-rank overdetermined least-squares problems?
• RQ2 Can the residual approximation in EK be replaced by coordinate descent without loss of convergence quality or efficiency?
• RQ3 Is there a computational advantage in using coordinate descent followed by standard Kaczmarz instead of EK for general least-squares problems?
• RQ4 How does the performance of EK compare to CD and hybrid CD+K methods in terms of residual error reduction over iterations?
• RQ5 Why does EK fail to outperform CD despite its dual use of column and row projections?

Key findings

• Coordinate descent converges in fewer iterations than the randomized extended Kaczmarz algorithm for full-rank overdetermined least-squares problems.
• CD requires fewer operations per iteration than EK because it only uses matrix columns, while EK uses both columns and rows.
• For full-rank overdetermined problems, CD is always preferable to EK due to faster convergence and lower computational cost.
• In rank-deficient or unknown-rank problems, CD followed by standard Kaczmarz (CD+K) is more efficient than EK and performs nearly identically to CD+EK+K.
• Numerical results show CD’s convergence is consistently faster than EK, with CD+K and CD+EK+K achieving similar performance to CD alone in early iterations.
• The advantage of CD over EK is most pronounced in early iterations, where EK’s residual approximation is poor, degrading Kaczmarz step quality.