[Paper Review] Random Osborne: a simple, practical algorithm for Matrix Balancing in near-linear time.

Jason M. Altschuler, Pablo A. Parrilo|arXiv (Cornell University)|Apr 6, 2020

Stochastic Gradient Optimization Techniques37 references4 citations

TL;DR

This paper introduces Random Osborne, a randomized variant of Osborne's algorithm for matrix balancing that achieves near-linear time convergence—specifically, $O(m heta^{-2} \log \kappa)$ operations for $\ell_1$ accuracy—by leveraging a convex optimization perspective and log-convexity of a Hellinger-distance-based potential function. The method improves upon prior theoretical bounds by ensuring practical $\ell_1$ accuracy and extending to cyclic, greedy, and parallel variants.

ABSTRACT

We revisit Matrix Balancing, a pre-conditioning task used ubiquitously for computing eigenvalues and matrix exponentials. Since 1960, Osborne's algorithm has been the practitioners' algorithm of choice, and is now implemented in most numerical software packages. However, the theoretical properties of Osborne's algorithm are not well understood. Here, we show that a simple random variant of Osborne's algorithm converges in near-linear time in the input sparsity. Specifically, it balances $K\in\mathbb{R}_{\geq 0}^{n imes n}$ after $O(m\epsilon^{-2}\log\kappa)$ arithmetic operations, where $m$ is the number of nonzeros in $K$, $\epsilon$ is the $\ell_1$ accuracy, and $\kappa=\sum_{ij}K_{ij}/(\min_{ij:K_{ij} eq 0}K_{ij})$ measures the conditioning of $K$. Previous work had established near-linear runtimes either only for $\ell_2$ accuracy (a weaker criterion which is less relevant for applications), or through an entirely different algorithm based on (currently) impractical Laplacian solvers. We further show that if the graph with adjacency matrix $K$ is moderately connected--e.g., if $K$ has at least one positive row/column pair--then Osborne's algorithm initially converges exponentially fast, yielding an improved runtime $O(m\epsilon^{-1}\log\kappa)$. We also address numerical precision issues by showing that these runtime bounds still hold when using $O(\log(n\kappa/\epsilon))$-bit numbers. Our results are established through a potential argument that leverages a convex optimization perspective of Osborne's algorithm, and relates the per-iteration progress to the current imbalance as measured in Hellinger distance. Unlike previous analyses, we critically exploit log-convexity of the potential. Our analysis extends to other variants of Osborne's algorithm: along the way, we establish significantly improved runtime bounds for cyclic, greedy, and parallelized variants.

Motivation & Objective

• To address the long-standing theoretical gap in understanding Osborne's algorithm, which remains the de facto standard in numerical software despite poor theoretical convergence guarantees.
• To develop a variant of Osborne's algorithm that achieves near-linear time convergence for $\ell_1$ accuracy, a more relevant criterion for practical applications than $\ell_2$ accuracy.
• To establish runtime bounds that hold under finite-precision arithmetic, using $O(\log(n\kappa/\epsilon))$-bit numbers, ensuring numerical feasibility.
• To extend theoretical convergence guarantees to cyclic, greedy, and parallelized variants of Osborne's algorithm through a unified potential function analysis.

Proposed method

• Propose a randomized variant of Osborne's algorithm that selects rows/columns uniformly at random at each iteration, ensuring faster and more predictable convergence than the original deterministic version.
• Define a potential function based on Hellinger distance to measure imbalance, which is shown to be log-convex and enables strong per-iteration progress bounds.
• Use a convex optimization framework to relate the decrease in potential to the current imbalance, enabling a tight analysis of convergence speed.
• Leverage the log-convexity of the potential function to derive exponential convergence in early iterations when the matrix is moderately connected.
• Establish runtime bounds under finite-precision arithmetic by analyzing bit complexity, showing that $O(\log(n\kappa/\epsilon))$-bit numbers suffice to maintain theoretical guarantees.
• Extend the analysis to cyclic, greedy, and parallel variants by adapting the potential function argument to their respective update rules.

Experimental results

Research questions

• RQ1 Can a randomized variant of Osborne's algorithm achieve near-linear time convergence for $\ell_1$ accuracy, which is more relevant for numerical applications than $\ell_2$ accuracy?
• RQ2 Does the convergence rate of Osborne's algorithm improve significantly when the underlying matrix graph is moderately connected, such as having at least one positive row/column pair?
• RQ3 Can the theoretical runtime bounds of Osborne's algorithm be preserved under finite-precision arithmetic using only $O(\log(n\kappa/\epsilon))$ bits?
• RQ4 Can the same potential function and analysis framework be extended to other variants of Osborne's algorithm, such as cyclic, greedy, and parallelized versions?

Key findings

• Random Osborne converges in $O(m\epsilon^{-2}\log\kappa)$ arithmetic operations to $\ell_1$ accuracy, where $m$ is the number of nonzeros and $\kappa$ measures matrix conditioning.
• When the matrix graph is moderately connected, convergence becomes exponentially fast, reducing the runtime to $O(m\epsilon^{-1}\log\kappa)$.
• The theoretical runtime bounds hold under finite-precision arithmetic using only $O(\log(n\kappa/\epsilon))$-bit numbers, ensuring practical implementability.
• The analysis via a log-convex potential function based on Hellinger distance provides a new, more powerful framework for analyzing Osborne-type algorithms.
• The same theoretical guarantees are extended to cyclic, greedy, and parallelized variants, significantly improving their known runtime bounds.
• This work resolves a long-standing theoretical gap by providing the first near-linear time analysis for $\ell_1$ accuracy in matrix balancing using a practical algorithm.