[Paper Review] On Randomized Distributed Coordinate Descent with Quantized Updates

Mostafa El Gamal, Lifeng Lai|arXiv (Cornell University)|Sep 18, 2016

Stochastic Gradient Optimization Techniques9 references3 citations

TL;DR

This paper proposes a quantized variant of the randomized distributed coordinate descent algorithm for convex optimization under finite-precision communication. It establishes sufficient conditions on quantization error magnitude to guarantee convergence to the optimal solution, showing that convergence is preserved as long as quantization error is bounded relative to problem parameters and desired accuracy.

ABSTRACT

In this paper, we study the randomized distributed coordinate descent algorithm with quantized updates. In the literature, the iteration complexity of the randomized distributed coordinate descent algorithm has been characterized under the assumption that machines can exchange updates with an infinite precision. We consider a practical scenario in which the messages exchange occurs over channels with finite capacity, and hence the updates have to be quantized. We derive sufficient conditions on the quantization error such that the algorithm with quantized update still converge. We further verify our theoretical results by running an experiment, where we apply the algorithm with quantized updates to solve a linear regression problem.

Motivation & Objective

• To address the gap in convergence analysis of distributed optimization algorithms under finite-capacity communication channels.
• To investigate whether randomized coordinate descent remains convergent when gradient updates are quantized due to limited channel capacity.
• To derive explicit sufficient conditions on quantization error that preserve convergence to the optimal solution.
• To validate theoretical findings through empirical evaluation on a linear regression problem with controlled quantization levels.

Proposed method

• Adapts the randomized coordinate descent algorithm to a distributed setting with quantized message exchanges over finite-capacity channels.
• Models the quantization error as a bounded perturbation in the update direction, with quantization resolution Δ.
• Derives a convergence rate bound using strong convexity and Lipschitz smoothness parameters (m and L), and condition number g = L/m.
• Introduces a modified iteration count k^q that accounts for quantization-induced error accumulation in the convergence analysis.
• Uses a step size t_opt = 1/(gLd) to minimize the convergence rate constant C_min = 1 - 1/(g²d).
• Derives a sufficient condition Δ ≤ (ερL²)/(2m) * (1/C_min - 1) for convergence within ε accuracy with confidence ρ.

Experimental results

Research questions

• RQ1 Under what conditions does randomized distributed coordinate descent converge when updates are quantized due to finite-capacity communication channels?
• RQ2 How does quantization error affect the convergence rate of distributed coordinate descent algorithms?
• RQ3 Can the convergence of the algorithm be preserved under bounded quantization error, and if so, what is the maximum allowable error?
• RQ4 What is the relationship between quantization resolution, problem parameters (L, m, d), and the required number of iterations for convergence?
• RQ5 Does quantization error propagate and destabilize the algorithm, or can it be controlled to ensure convergence?

Key findings

• The algorithm converges to the optimal solution if the quantization error Δ is bounded above by (ερL²)/(2m) * (1/C_min - 1), where ε is the desired accuracy and ρ is the confidence level.
• The number of iterations required for convergence is bounded by k^q = log(2||x₀ - x*||²/ερ)/log(1/C_min) + log(2||x₀ - x*||²)/log(1/(C_min + (ερ/2)(1 - C_min))), which accounts for quantization-induced error.
• When Δ = 0 (i.e., no quantization), the iteration count reduces to k = log(||x₀ - x*||²/ερ)/log(1/C_min), matching the known result for the quantization-free case.
• Empirical results on a power plant dataset show that with Δ = 10³, the algorithm converges, but with Δ = 10⁵, it diverges, confirming the theoretical bound on allowable error.
• The convergence is robust to quantization as long as the error is sufficiently small, and quantization error does not propagate to destabilize the algorithm.
• The derived convergence condition ensures that the algorithm maintains a linear convergence rate even under quantization, provided the error is within the derived threshold.