[Paper Review] Maximum Likelihood for Matrices with Rank Constraints

Jonathan D. Hauenstein, Jose Israel Rodriguez+1 more|arXiv (Cornell University)|Sep 30, 2012

Bayesian Methods and Mixture Models4 citations

TL;DR

This paper develops numerical algebraic geometry methods to compute all critical points of the likelihood function on determinantal varieties of matrices with bounded rank, enabling reliable global maximum likelihood estimation. The key contribution is the exact computation of maximum likelihood degrees for rank-constrained matrices, revealing a duality between complementary ranks and resolving long-standing open cases in algebraic statistics.

ABSTRACT

Maximum likelihood estimation is a fundamental optimization problem in statistics. We study this problem on manifolds of matrices with bounded rank. These represent mixtures of distributions of two independent discrete random variables. We determine the maximum likelihood degree for a range of determinantal varieties, and we apply numerical algebraic geometry to compute all critical points of their likelihood functions. This led to the discovery of maximum likelihood duality between matrices of complementary ranks, a result proved subsequently by Draisma and Rodriguez.

Motivation & Objective

• To compute all critical points of the likelihood function on determinantal varieties of matrices with bounded rank, which model statistical mixtures of independent discrete random variables.
• To address the challenge of multiple local maxima in maximum likelihood estimation (MLE) for low-rank matrix models, where global optimality is hard to verify.
• To determine the maximum likelihood degree (ML degree) for determinantal varieties of rank ≤ r in m×n matrices, especially for cases beyond symbolic computation limits.
• To establish and verify a duality between ML degrees for matrices of complementary ranks, such as r and m−r+1.
• To provide a computational framework that reliably identifies all local maxima of the likelihood function in low-rank matrix models, enabling global optimization.

Proposed method

• Formulates the MLE problem on rank-constrained matrices as a square system of polynomial equations using a novel parametrization involving rank-r matrices via factorization P = AB^T.
• Applies numerical algebraic geometry techniques, including homotopy continuation and numerical irreducible decomposition, to compute all complex critical points of the likelihood function on the determinantal variety.
• Uses the likelihood equations derived from the Lagrangian of the log-likelihood function under rank constraints, leading to a system of polynomial equations in the entries of A and B.
• Employs the EM algorithm as a numerical benchmark to validate results, comparing its convergence to the computed critical points and identifying boundary solutions.
• Leverages symmetry and duality in the likelihood equations to reduce computational complexity and verify conjectures about complementary rank duality.
• Validates results against known cases and uses high-precision numerical solvers to confirm the exact count of critical points for various (m,n,r) configurations.

Experimental results

Research questions

• RQ1 What is the exact number of critical points (i.e., the ML degree) of the likelihood function on the variety of m×n matrices of rank ≤ r?
• RQ2 Does a duality exist between the ML degrees of matrices of rank r and rank m−r+1 for m≤n?
• RQ3 Can numerical algebraic geometry methods reliably compute all critical points of the likelihood function in high-dimensional, low-rank matrix models where symbolic computation fails?
• RQ4 How do the EM algorithm and numerical continuation methods compare in identifying local maxima in rank-constrained MLE problems?
• RQ5 To what extent do the critical points of the likelihood function lie in the interior of the parameter space, and when do they lie on the boundary?

Key findings

• The ML degree for 4×4 matrices of rank ≤2 is 191, resolving a long-standing open problem previously inaccessible to symbolic computation.
• The ML degree for 5×5 matrices of rank ≤3 is 61326, and for rank ≤2 it is 3119, confirming the first non-trivial values in this size class.
• The ML degree for 4×6 matrices of rank ≤3 is 3119, and for 5×5 matrices of rank ≤4 is 6776, demonstrating the symmetry between complementary ranks.
• The paper confirms the duality between ML degrees for ranks r and m−r+1, a conjecture proven later by Draisma and Rodriguez, by computing explicit values.
• For 4×5 matrices of rank ≤3, 10,000 EM algorithm runs identified 10 local maxima, of which 8 were confirmed to be the 8 critical points computed via numerical algebraic geometry.
• Two additional local maxima found by the EM algorithm lie on the boundary of the parameter space and do not satisfy the likelihood equations, indicating they are not critical points in the interior.