[Paper Review] Distributed coordinate descent methods for composite minimization

Ion Necoara, Dragos Clipici|arXiv (Cornell University)|Dec 18, 2013

Stochastic Gradient Optimization Techniques8 citations

TL;DR

This paper proposes a distributed randomized block-coordinate descent method for composite minimization involving a partially separable smooth convex function and a fully separable non-smooth convex function. Under block Lipschitz gradient assumptions, the method achieves a sublinear convergence rate, with linear convergence for a new class of generalized error bound functions that include strongly convex and error-bound functions, where convergence depends on block selection and function separability.

ABSTRACT

In this paper we propose a distributed version of a randomized block-coordinate descent method for minimizing the sum of a partially separable smooth convex function and a fully separable non-smooth convex function. Under the assumption of block Lipschitz continuity of the gradient of the smooth function, this method is shown to have a sublinear convergence rate. Linear convergence rate of the method is obtained for the newly introduced class of generalized error bound functions. We prove that the new class of generalized error bound functions encompasses both global/local error bound functions and smooth strongly convex functions. We also show that the theoretical estimates on the convergence rate depend on the number of blocks chosen randomly and a natural measure of separability of the objective function.

Motivation & Objective

• To develop a distributed optimization method for composite convex problems with separable structures.
• To analyze convergence rates under block Lipschitz gradient assumptions.
• To establish linear convergence for a new class of generalized error bound functions.
• To quantify the impact of block selection and function separability on convergence speed.

Proposed method

• The method employs randomized block-coordinate descent in a distributed setting for minimizing the sum of a smooth partially separable function and a non-smooth fully separable function.
• It leverages block-wise gradient computations with updates performed in parallel across distributed nodes.
• Convergence analysis is based on block Lipschitz continuity of the gradient of the smooth component.
• The method introduces a new class of generalized error bound functions to establish linear convergence.
• Theoretical convergence rates are derived based on the number of randomly selected blocks and a measure of function separability.
• The approach is designed for large-scale problems where full gradient computation is infeasible.

Experimental results

Research questions

• RQ1 Can a distributed block-coordinate descent method achieve sublinear convergence for composite minimization with separable structures?
• RQ2 Does the method achieve linear convergence under a broader class of functions than previously known?
• RQ3 How does the number of randomly selected blocks affect the convergence rate?
• RQ4 What role does the separability measure of the objective function play in convergence speed?
• RQ5 Can the generalized error bound class unify existing convergence results for strongly convex and error-bound functions?

Key findings

• The method achieves a sublinear convergence rate under the assumption of block Lipschitz continuity of the gradient of the smooth function.
• Linear convergence is established for a new class of generalized error bound functions that include both global/local error bounds and smooth strongly convex functions.
• The convergence rate depends on the number of randomly selected blocks and a natural measure of separability of the objective function.
• The generalized error bound class provides a unifying framework that extends existing convergence results.
• Theoretical estimates show that convergence speed improves with higher separability and optimal block selection.