[Paper Review] Hybrid Jacobian and Gauss-Seidel proximal block coordinate update methods for linearly constrained convex programming

Yangyang Xu|arXiv (Cornell University)|Aug 13, 2016

Sparse and Compressive Sensing Techniques45 references3 citations

TL;DR

This paper proposes a hybrid Jacobian and Gauss-Seidel block coordinate update method for linearly constrained convex optimization with nonseparable quadratic terms. By approximating the augmented Lagrangian at affine combinations of previous iterates and solving a semidefinite program to select an optimal mixing matrix, the method achieves global convergence with sublinear rate while outperforming pure Jacobian and randomized methods in practice.

ABSTRACT

Recent years have witnessed the rapid development of block coordinate update (BCU) methods, which are particularly suitable for problems involving large-sized data and/or variables. In optimization, BCU first appears as the coordinate descent method that works well for smooth problems or those with separable nonsmooth terms and/or separable constraints. As nonseparable constraints exist, BCU can be applied under primal-dual settings. In the literature, it has been shown that for weakly convex problems with nonseparable linear constraint, BCU with fully Gauss-Seidel updating rule may fail to converge and that with fully Jacobian rule can converge sublinearly. However, empirically the method with Jacobian update is usually slower than that with Gauss-Seidel rule. To maintain their advantages, we propose a hybrid Jacobian and Gauss-Seidel BCU method for solving linearly constrained multi-block structured convex programming, where the objective may have a nonseparable quadratic term and separable nonsmooth terms. At each primal block variable update, the method approximates the augmented Lagrangian function at an affine combination of the previous two iterates, and the affinely mixing matrix with desired nice properties can be chosen through solving a semidefinite programming. We show that the hybrid method enjoys the theoretical convergence guarantee as Jacobian BCU. In addition, we numerically demonstrate that the method can perform as well as Gauss-Seidel method and better than a recently proposed randomized primal-dual BCU method.

Motivation & Objective

• Address the lack of reliable and efficient algorithms for solving large-scale linearly constrained multi-block convex problems with nonseparable quadratic terms.
• Overcome the limitations of pure Jacobian rule (slow convergence) and Gauss-Seidel rule (potential non-convergence) in block coordinate updates.
• Develop a hybrid method that retains the convergence guarantees of the Jacobian rule while achieving the practical efficiency of the Gauss-Seidel rule.
• Provide a systematic way to construct a mixing matrix via semidefinite programming that ensures convergence and improves performance.
• Demonstrate the method's effectiveness on real-world problems such as nonnegative linearly constrained quadratic programming and multiclass support vector machines.

Proposed method

• Formulate the problem as a multi-block convex program with a nonseparable quadratic term and separable nonsmooth functions, subject to linear equality constraints.
• Apply a block coordinate update strategy to the augmented Lagrangian function, updating one block at a time while fixing others.
• At each update, approximate the augmented Lagrangian at an affine combination of the current and previous iterates using a mixing matrix W.
• Construct the mixing matrix W by solving a semidefinite program to ensure desirable convergence properties and stability.
• Use a proximal linearization technique to simplify subproblem solutions, especially for non-smooth components.
• Update the dual variable (Lagrangian multiplier) using standard augmented Lagrangian steps after each primal update.

Experimental results

Research questions

• RQ1 Can a hybrid block coordinate method combining Jacobian and Gauss-Seidel rules achieve both global convergence and fast practical performance?
• RQ2 How can the mixing matrix be constructed to ensure convergence while balancing the trade-off between speed and stability?
• RQ3 Does the proposed method outperform existing randomized and fully Jacobian-based block coordinate methods in terms of convergence rate and feasibility?
• RQ4 Can the method be applied effectively to large-scale problems such as multiclass support vector machines and nonnegative quadratic programs?
• RQ5 What is the impact of the initial mixing parameter and adaptive step size on convergence behavior?

Key findings

• The hybrid method achieves global convergence with a sublinear convergence rate, matching the theoretical guarantees of the pure Jacobian rule.
• Numerical experiments show that the hybrid method performs as well as the Gauss-Seidel method in practice and significantly better than the randomized primal-dual BCU method.
• The method with adaptive parameter selection (d¹ = 0.5) converges faster and maintains better feasibility compared to the method with d¹ = 0.1.
• The mixing matrix W computed via semidefinite programming ensures stable and efficient iterates, with W having entries that reflect optimal weighting between previous iterates.
• For the multiclass SVM problem, JaGS-PC (hybrid method) achieves lower objective gap and feasibility violation than Jacobi-PC, especially in early iterations.
• The running time of the hybrid method includes the cost of solving the SDP for W, but the overall performance gain justifies the overhead in large-scale settings.