[Paper Review] Linearly Convergent Randomized Iterative Methods for Computing the Pseudoinverse

Robert M. Gower, Peter Richtárik|arXiv (Cornell University)|Dec 19, 2016

Sparse and Compressive Sensing Techniques26 references17 citations

TL;DR

This paper introduces the first linearly convergent randomized iterative methods for computing the Moore-Penrose pseudoinverse of real matrices, leveraging three variational characterizations of the pseudoinverse. The proposed SATAX and SAXAS methods achieve linear convergence for general and symmetric matrices, respectively, and outperform the Newton-Schulz method on large-scale problems, especially in early iterations.

ABSTRACT

We develop the first stochastic incremental method for calculating the Moore-Penrose pseudoinverse of a real matrix. By leveraging three alternative characterizations of pseudoinverse matrices, we design three methods for calculating the pseudoinverse: two general purpose methods and one specialized to symmetric matrices. The two general purpose methods are proven to converge linearly to the pseudoinverse of any given matrix. For calculating the pseudoinverse of full rank matrices we present two additional specialized methods which enjoy a faster convergence rate than the general purpose methods. We also indicate how to develop randomized methods for calculating approximate range space projections, a much needed tool in inexact Newton type methods or quadratic solvers when linear constraints are present. Finally, we present numerical experiments of our general purpose methods for calculating pseudoinverses and show that our methods greatly outperform the Newton-Schulz method on large dimensional matrices.

Motivation & Objective

• Develop the first stochastic incremental methods for computing the Moore-Penrose pseudoinverse with provable linear convergence.
• Address the limitations of SVD and Newton-Schulz methods in big data settings, where memory and computational costs are prohibitive.
• Design specialized methods for symmetric matrices to improve convergence speed and efficiency.
• Enable efficient approximation of range space projections, useful in inexact Newton methods and quadratic solvers with constraints.
• Combine the strengths of randomized methods and Newton-Schulz for hybrid algorithms that outperform either method alone.

Proposed method

• Leverage three variational characterizations of the pseudoinverse (P1, P2, P3) to derive iterative update rules based on minimizing Frobenius norm solutions.
• Propose SATAX (Stochastic Averaged X) for general matrices using sketching and projection onto randomized subspaces to update the pseudoinverse estimate.
• Introduce SAXAS (Stochastic Averaged X for Symmetric matrices) as a specialized method for symmetric matrices, using symmetric sketches and tailored update rules.
• Use randomized sketching matrices S to compute low-rank approximations of matrix products (e.g., AS, A^TAS), reducing computational cost per iteration.
• Apply a normalization heuristic to SATAX iterates before switching to Newton-Schulz to satisfy its convergence condition.
• Design hybrid NS-SATAX and NS-SAXAS methods that combine early-stage randomized convergence with local quadratic convergence of Newton-Schulz.

Experimental results

Research questions

• RQ1 Can randomized incremental methods be designed to compute the pseudoinverse with linear convergence rates?
• RQ2 How can the pseudoinverse be computed efficiently for large-scale matrices where SVD and Newton-Schulz methods fail due to memory or time constraints?
• RQ3 Can specialized methods for symmetric matrices achieve faster convergence than general-purpose randomized methods?
• RQ4 What is the optimal strategy for combining randomized methods with Newton-Schulz to achieve superior overall performance?
• RQ5 Can randomized methods be extended to approximate range space projections, enabling applications in constrained optimization?

Key findings

• The SATAX and SAXAS methods converge linearly to the pseudoinverse for any real matrix and symmetric matrix, respectively, with theoretical convergence guarantees.
• SATAX outperforms the Newton-Schulz method on large-dimensional matrices, especially in early iterations, due to faster initial convergence.
• The hybrid NS-SATAX method, which switches from SATAX to Newton-Schulz after one effective pass, achieves better overall performance than Newton-Schulz alone.
• For symmetric matrices, SAXAS_uni and SAXAS_ada significantly outperform Newton-Schulz in reaching relative residuals below 10^-6 on real-world datasets like a9a and gisette_scale.
• On randomly generated Gaussian matrices, Newton-Schulz remains more efficient for high-precision pseudoinverse computation, justifying the hybrid approach.
• The proposed methods enable efficient approximation of range space projections, a key tool in inexact Newton methods and quadratic solvers with linear constraints.