[Paper Review] Coordinate Descent with Arbitrary Sampling II: Expected Separable Overapproximation

Zheng Qu, Peter Richtárik|arXiv (Cornell University)|Dec 27, 2014

Stochastic Gradient Optimization Techniques25 references16 citations

TL;DR

This paper develops a general framework for deriving Expected Separable Overapproximation (ESO) inequalities for arbitrary samplings in randomized coordinate descent methods. By analyzing eigenvalues of the Hadamard product between sampling probability matrices and data-dependent matrices, the authors establish ESO bounds for a broad class of functions, enabling improved convergence complexity analysis and optimal sampling design for both serial and parallel settings.

ABSTRACT

The design and complexity analysis of randomized coordinate descent methods, and in particular of variants which update a random subset (sampling) of coordinates in each iteration, depends on the notion of expected separable overapproximation (ESO). This refers to an inequality involving the objective function and the sampling, capturing in a compact way certain smoothness properties of the function in a random subspace spanned by the sampled coordinates. ESO inequalities were previously established for special classes of samplings only, almost invariably for uniform samplings. In this paper we develop a systematic technique for deriving these inequalities for a large class of functions and for arbitrary samplings. We demonstrate that one can recover existing ESO results using our general approach, which is based on the study of eigenvalues associated with samplings and the data describing the function.

Motivation & Objective

• To establish a systematic method for deriving Expected Separable Overapproximation (ESO) inequalities for arbitrary samplings, not limited to uniform or i.i.d. cases.
• To unify and generalize existing ESO results in the literature by providing a common framework based on eigenvalue analysis of sampling and data matrices.
• To enable tighter convergence complexity bounds for randomized coordinate descent methods by deriving optimal sampling probabilities tailored to the problem structure.
• To support the design of accelerated and parallel coordinate descent methods with provable convergence guarantees under general sampling schemes.
• To demonstrate the practical impact of improved ESO parameters through complexity comparisons on real datasets like dorothea and w8a.

Proposed method

• The method is based on analyzing the largest and normalized largest eigenvalues of the Hadamard product between the sampling probability matrix $ P $ and a data-dependent positive semidefinite matrix $ A^ op A $, where $ A $ encodes the function's structure.
• For quadratic functions $ f(x) = \|Ax\|^2 $, the ESO inequality reduces to an eigenvalue problem: $ \mathbb{E}[\|\sum_{i\in\hat{S}} A_i h_i\|^2] \leq h^\top \operatorname{Diag}(p \circ v) h $, with $ v $ derived from the spectral properties of $ P \circ A^\top A $.
• The authors derive general ESO bounds using the spectral norm of $ P \circ A^\top A $, leading to explicit formulas for $ v_i $, such as $ v_i = \max_i \|A_i\|^2 \cdot \frac{\tau}{n} $ for uniform sampling, and more refined expressions for non-uniform samplings.
• A key technique involves expressing the ESO parameter $ v $ in terms of the maximum eigenvalue of $ P \circ A^\top A $, enabling the derivation of complexity bounds in terms of $ \|d\|_q $ norms.
• The framework allows for the derivation of optimal sampling probabilities $ p_i $ that minimize the condition number in the complexity bound, leading to $ p_i \propto (w_i (x_i^0 - x_i^*)^2)^{1/3} $ for serial methods.
• The approach is extended to block coordinate descent by replacing coordinates with blocks and adjusting the norm and matrix structure accordingly.

Experimental results

Research questions

• RQ1 Can ESO inequalities be systematically derived for arbitrary samplings beyond uniform or i.i.d. cases?
• RQ2 What is the role of the spectral properties of the sampling-probability matrix and data matrix in determining the ESO parameter $ v $?
• RQ3 How can optimal sampling probabilities be derived to minimize the iteration complexity of randomized coordinate descent?
• RQ4 Can existing ESO results for uniform samplings be re-derived using this general eigenvalue-based framework?
• RQ5 What is the trade-off between the computational cost of computing a refined $ v $ and the resulting reduction in iteration complexity?

Key findings

• The paper establishes a general method to derive ESO inequalities for arbitrary samplings by analyzing the largest eigenvalue of the Hadamard product $ P \circ A^\top A $, where $ P $ is the probability matrix of the sampling.
• For the dorothea dataset with $ \tau = 256 $, using Formula 60 to compute $ v $ reduces the iteration complexity to $ O(256.91 + \frac{256.91}{\lambda} \log(\frac{1}{\epsilon})) $, compared to $ O(8715.8 + \frac{16.68}{\lambda} \log(\frac{1}{\epsilon})) $ using Formula 58, showing a significant improvement.
• The optimal serial sampling probabilities $ p_i \propto (w_i (x_i^0 - x_i^*)^2)^{1/3} $ lead to a complexity bound $ C_{\text{opt}} = \sqrt{2} \|d\|_2^3 / \sqrt{\epsilon} $, which can be up to $ n $ times better than uniform sampling when $ \|d\|_6 = \|d\|_2 $.
• For the w8a dataset, the time to compute $ v $ using Formula 60 is only 1.8× the time of one data pass, while reducing the condition number by a factor of ~380 compared to Formula 57.
• The framework recovers known ESO results for uniform samplings using a different, eigenvalue-based derivation, demonstrating its generality and unifying power.
• The method enables the design of accelerated coordinate descent methods with provably better convergence rates by using data-aware, non-uniform sampling strategies.