[Paper Review] Randomized Methods for Linear Constraints: Convergence Rates and Conditioning

D. Leventhal, Adrian S. Lewis|ArXiv.org|Jun 18, 2008

Sparse and Compressive Sensing Techniques18 references4 citations

TL;DR

This paper analyzes randomized variants of coordinate descent and iterated projection algorithms for solving linear systems and inequalities, showing that under appropriate probability distributions, convergence rates in expectation are bounded by condition numbers and metric regularity measures. The key contribution is a theoretical framework linking linear convergence rates to natural condition measures like the Hoffman error bound and distance to ill-posedness.

ABSTRACT

We study randomized variants of two classical algorithms: coordinate descent for systems of linear equations and iterated projections for systems of linear inequalities. Expanding on a recent randomized iterated projection algorithm of Strohmer and Vershynin for systems of linear equations, we show that, under appropriate probability distributions, the linear rates of convergence (in expectation) can be bounded in terms of natural linear-algebraic condition numbers for the problems. We relate these condition measures to distances to ill-posedness, and discuss generalizations to convex systems under metric regularity assumptions.

Motivation & Objective

• To establish linear convergence rates for randomized coordinate descent in solving linear systems using condition numbers.
• To extend randomized iterated projection methods to systems of linear inequalities and relate convergence to error bounds and distance to infeasibility.
• To generalize convergence analysis to convex feasibility problems under metric regularity assumptions.
• To connect classical conditioning measures—such as condition number and distance to ill-posedness—with randomized algorithm performance.
• To demonstrate that randomized algorithms can achieve convergence rates comparable to deterministic methods, but with simplified theoretical analysis.

Proposed method

• Uses randomized coordinate descent with probability distributions tailored to the structure of the linear system to ensure linear convergence in expectation.
• Applies a randomized iterated projection scheme where constraints are selected uniformly at random from a set of linear inequalities.
• Employs metric regularity and error bound theory to bound the expected decrease in distance to the solution set at each iteration.
• Utilizes Pierra’s product space formulation to reduce m-set feasibility to a two-set problem, enabling analysis of averaged projection methods.
• Derives convergence rates via expectation inequalities, leveraging convexity and Jensen’s inequality to compare randomized and averaged projection schemes.
• Relies on the Hoffman error bound and Renegar’s distance to infeasibility as condition measures to quantify convergence speed.

Experimental results

Research questions

• RQ1 Can randomized coordinate descent for linear systems achieve linear convergence rates bounded by standard condition numbers?
• RQ2 How do randomized projection methods for linear inequalities compare to deterministic ones in terms of convergence speed and dependence on problem conditioning?
• RQ3 Can metric regularity and error bounds be used to derive explicit convergence rates for randomized projection algorithms?
• RQ4 What is the relationship between the convergence rate of randomized projections and the distance to ill-posedness in linear feasibility problems?
• RQ5 Does the method of averaged projections converge at least as fast as uniformly randomized projections, and can this be quantified?

Key findings

• The randomized coordinate descent method converges linearly in expectation with a rate bounded by a function of the condition number of the system matrix.
• For systems of linear inequalities, the randomized iterated projection method achieves linear convergence in expectation with a rate related to the Hoffman error bound.
• The convergence rate of the randomized algorithm is bounded by $1 - \frac{1}{(m-1)\bar{\gamma}^2}$, where $\bar{\gamma}$ is a regularity modulus larger than the true modulus $\gamma$.
• The method of averaged projections converges no more slowly than the randomized projection method, with a rate bounded by $1 - \frac{1}{m\bar{\gamma}^2}$.
• The analysis links classical condition measures—such as the distance to ill-posedness and the Eckart-Young theorem—to algorithmic convergence rates in randomized settings.
• The results generalize to convex feasibility problems under metric regularity, showing that convergence rates depend on the modulus of regularity.