[Paper Review] Block Coordinate Descent Only Converge to Minimizers

Enbin Song, Zhubin Shen+1 more|arXiv (Cornell University)|Oct 25, 2017

Stochastic Gradient Optimization Techniques18 references3 citations

TL;DR

This paper proves that block coordinate gradient descent, block mirror descent, and proximal block coordinate descent converge to local minimizers almost surely with random initialization, even for non-convex functions with non-isolated critical points. The result extends prior work on gradient descent by leveraging topological arguments and complex analysis to show convergence despite saddle points and non-unique critical points.

ABSTRACT

Given a non-convex twice continuously differentiable cost function with Lipschitz continuous gradient, we prove that all of block coordinate gradient descent, block mirror descent and proximal block coordinate descent converge to a local minimizer, almost surely with random initialization. Furthermore, we show that these results also hold true even for the cost functions with non-isolated critical points.

Motivation & Objective

• To establish convergence to local minimizers for block coordinate descent variants in non-convex optimization under mild smoothness conditions.
• To resolve open questions on whether block coordinate methods avoid saddle points, as posed in Lee et al. (2016).
• To extend the almost-sure convergence result to non-isolated critical points, generalizing prior work by Panageas and Piliouras (2016).
• To provide a deterministic analysis without requiring isotropic noise, unlike prior stochastic approaches.
• To unify convergence guarantees across three major block-coordinate frameworks: gradient, mirror, and proximal descent.

Proposed method

• Prove that the iterative mapping of each block coordinate method is a diffeomorphism using the chain rule and block-wise differentiability.
• Transform the Jacobian of the iterative map at a critical point into a simplified form via eigen-decomposition and block matrix manipulation.
• Apply Rouche’s Theorem from complex analysis to show that the Jacobian has at least one eigenvalue with magnitude strictly greater than one at saddle points.
• Use the strong convexity of the potential function in mirror and proximal methods to ensure the gradient mapping is a diffeomorphism.
• Establish that the convergence to local minimizers holds under the assumption of twice continuously differentiable cost functions with Lipschitz continuous gradients.
• Handle non-isolated critical points by leveraging the countable subcover property of open covers in R^n, extending prior results.

Experimental results

Research questions

• RQ1 Do block coordinate descent methods converge to local minimizers almost surely with random initialization in non-convex settings?
• RQ2 Can the convergence guarantee be extended to functions with non-isolated critical points?
• RQ3 Do block mirror descent and proximal block coordinate descent also avoid saddle points and converge to local minimizers?
• RQ4 Can deterministic block-coordinate methods achieve almost-sure convergence to minimizers without requiring isotropic noise?
• RQ5 Is the Jacobian of the block update mapping at a saddle point guaranteed to have an eigenvalue of magnitude greater than one, ensuring instability at non-minimizers?

Key findings

• All three block coordinate methods—gradient, mirror, and proximal—converge to a local minimizer almost surely with random initialization.
• The convergence result holds even when the cost function has non-isolated critical points, generalizing earlier results that required isolated critical points.
• The Jacobian of the iterative map at a saddle point has at least one eigenvalue with magnitude strictly greater than one, ensuring instability at non-minimizers.
• The proof relies on transforming the Jacobian into a block matrix form and applying Rouche’s Theorem to analyze eigenvalue magnitude.
• The gradient mapping in mirror and proximal methods is a diffeomorphism due to strong convexity, ensuring smooth invertibility.
• The analysis avoids stochastic noise and applies to constant step-sizes, making the results applicable to standard implementations.