[Paper Review] Beyond Moore-Penrose Part I: Generalized Inverses that Minimize Matrix Norms

Ivan Dokmanić, Rémi Gribonval|arXiv (Cornell University)|Jun 26, 2017

Sparse and Compressive Sensing TechniquesEngineering27 references16 citations

TL;DR

This paper introduces a framework for generalized inverses that minimize various matrix norms beyond the Frobenius norm, demonstrating that the Moore-Penrose pseudoinverse (MPP) minimizes a broader class of unitarily invariant norms. It identifies conditions under which alternative inverses—particularly those minimizing mixed norms and ℓ^p→ℓ^q norms—yield superior performance in sparse recovery and inverse problems, with the key result that MPP is the unique norm-minimizing inverse only for p=2 in certain settings.

ABSTRACT

This is the first paper of a two-long series in which we study linear generalized inverses that minimize matrix norms. Such generalized inverses are famously represented by the Moore-Penrose pseudoinverse (MPP) which happens to minimize the Frobenius norm. Freeing up the degrees of freedom associated with Frobenius optimality enables us to promote other interesting properties. In this Part I, we look at the basic properties of norm-minimizing generalized inverses, especially in terms of uniqueness and relation to the MPP. We first show that the MPP minimizes many norms beyond those unitarily invariant, thus further bolstering its role as a robust choice in many situations. We then concentrate on some norms which are generally not minimized by the MPP, but whose minimization is relevant for linear inverse problems and sparse representations. In particular, we look at mixed norms and the induced $\ell^p ightarrow \ell^q$ norms. An interesting representative is the sparse pseudoinverse which we study in much more detail in Part II. Next, we shift attention from norms to matrices with interesting behaviors. We exhibit a class whose generalized inverse is always the MPP-even for norms that normally result in different inverses-and a class for which many generalized inverses coincide, but not with the MPP. Finally, we discuss efficient computation of norm-minimizing generalized inverses.

Motivation & Objective

• To extend the understanding of generalized inverses beyond the Moore-Penrose pseudoinverse by studying those that minimize various matrix norms.
• To identify conditions under which non-MPP inverses minimize norms such as mixed norms and ℓ^p→ℓ^q norms, which are relevant for sparse representations.
• To characterize matrix classes for which the MPP is the unique norm-minimizing inverse, and others where multiple inverses coincide without being the MPP.
• To establish theoretical foundations for efficient computation of norm-minimizing generalized inverses.
• To lay the groundwork for Part II, which investigates the sparse pseudoinverse in detail.

Proposed method

• The paper analyzes the uniqueness and structure of generalized inverses that minimize arbitrary matrix norms, focusing on unitarily invariant and non-unitarily invariant norms.
• It derives conditions under which the Moore-Penrose pseudoinverse minimizes norms beyond the Frobenius norm, using geometric and variational arguments.
• It introduces and studies the sparse pseudoinverse as a norm-minimizing inverse that minimizes the ℓ^1 norm of the vectorized inverse matrix under the constraint AX=I.
• It constructs explicit counterexamples of matrices for which the MPP does not minimize certain ℓ^p→ℓ^q norms, demonstrating the existence of distinct optimal inverses.
• It uses analytical techniques, including convexity arguments and derivative analysis, to prove that the equation for norm minimization vanishes only at p=2 for specific matrix constructions.
• It establishes a connection between norm minimization and the structure of projection matrices, particularly in rank-deficient settings.

Experimental results

Research questions

• RQ1 Under what conditions does the Moore-Penrose pseudoinverse minimize matrix norms beyond the Frobenius norm?
• RQ2 For which matrix norms other than unitarily invariant ones is the Moore-Penrose pseudoinverse not optimal, and what are the resulting generalized inverses?
• RQ3 Can generalized inverses that minimize ℓ^p→ℓ^q norms or mixed norms be efficiently computed and yield better performance in sparse recovery?
• RQ4 Are there matrix classes for which the Moore-Penrose pseudoinverse is the unique norm-minimizing inverse, and others where multiple inverses coincide without being the MPP?
• RQ5 What is the precise condition under which the MPP minimizes the ℓ^p→ℓ^q norm for a given matrix, and when does this fail?

Key findings

• The Moore-Penrose pseudoinverse minimizes a broad class of unitarily invariant norms, reinforcing its role as a robust choice in matrix inversion.
• For ℓ^p→ℓ^q norms with p≠2, the Moore-Penrose pseudoinverse is not the minimizer, and alternative inverses exist that are optimal for such norms.
• The sparse pseudoinverse, defined as the solution to minimizing the ℓ^1 norm of the vectorized inverse under AX=I, is shown to yield well-conditioned and sparse inverses, unlike simple submatrix inversion.
• There exist matrices for which the Moore-Penrose pseudoinverse is the unique norm-minimizing inverse, even for non-unitarily invariant norms.
• For a specific 3×3 matrix construction, the MPP is the unique minimizer of the ℓ^p→ℓ^q norm only when p=2, and not for p>2 or p<2.
• The paper proves analytically that the equation for norm minimality in a constructed example vanishes only at p=2, confirming that MPP is optimal in this case precisely when p=2.