[Paper Review] On the Provable Convergence of Alternating Minimization for Matrix Completion.

Moritz Hardt|arXiv (Cornell University)|Dec 3, 2013

Sparse and Compressive Sensing Techniques19 references14 citations

TL;DR

This paper presents a provably convergent alternating minimization algorithm for matrix completion that achieves linear convergence under standard incoherence assumptions, reducing required sample complexity by at least a quartic factor in rank and condition number compared to prior work. The method leverages a robust analysis of subspace iteration and introduces a novel coherence control technique for intermediate iterates.

ABSTRACT

Alternating Minimization is a widely used and empirically successful framework for Matrix Completion and related low-rank optimization problems. We give a new algorithm based on Alternating Minimization that provably recovers an unknown low-rank matrix from a random subsample of its entries under a standard incoherence assumption while achieving a linear convergence rate. Compared to previous work our results reduce the provable sample complexity requirements of the Alternating Minimization approach by at least a quartic factor in the rank and the condition number of the unknown matrix. These improvements apply when the matrix is exactly low-rank and when it is only close to low-rank in the Frobenius norm. Underlying our work is a new robust convergence analysis of the well-known Subspace Iteration algorithm for computing the dominant singular vectors of a matrix also known as the Power Method. This viewpoint leads to a conceptually simple understanding of Alternating Minimization that we exploit. Additionally, we contribute a new technique for controlling the coherence of intermediate solutions arising in iterative algorithms. These techniques may be of interest beyond their application here.

Motivation & Objective

• To establish provable convergence guarantees for alternating minimization in low-rank matrix completion.
• To reduce the required sample complexity for recovery of low-rank matrices by at least a quartic factor in rank and condition number.
• To extend convergence guarantees to matrices that are only approximately low-rank in Frobenius norm.
• To develop a robust analysis of subspace iteration for handling noisy or incomplete data in iterative low-rank approximation.
• To introduce a new technique for controlling coherence of intermediate solutions in iterative matrix recovery algorithms.

Proposed method

• The algorithm employs alternating minimization to iteratively estimate row and column spaces of the unknown low-rank matrix from a random subset of observed entries.
• It leverages a robust convergence analysis of the subspace iteration (power method) for computing dominant singular subspaces, enabling stability under incomplete data.
• A novel coherence control mechanism is introduced to bound the coherence of intermediate solution matrices, preventing ill-conditioning during iterations.
• The analysis establishes linear convergence rates under standard incoherence assumptions on the unknown matrix.
• Theoretical guarantees are derived for both exactly low-rank and approximately low-rank matrices in the Frobenius norm.
• The method achieves improved sample complexity by reducing dependence on rank and condition number by a quartic factor compared to prior provable results.

Experimental results

Research questions

• RQ1 Can alternating minimization for matrix completion be proven to converge linearly under standard incoherence assumptions with reduced sample complexity?
• RQ2 How can the coherence of intermediate solutions in iterative low-rank algorithms be controlled to ensure stable convergence?
• RQ3 What is the minimal sample complexity required for provable recovery of low-rank matrices using alternating minimization?
• RQ4 Can the convergence analysis of subspace iteration be extended to handle incomplete or noisy data in matrix completion?
• RQ5 How do the theoretical guarantees extend to matrices that are only approximately low-rank?

Key findings

• The proposed algorithm achieves linear convergence for matrix completion under standard incoherence assumptions.
• The required sample complexity is reduced by at least a quartic factor in both rank and condition number compared to prior provable results.
• The method applies to both exactly low-rank and approximately low-rank matrices in the Frobenius norm.
• A new robust analysis of subspace iteration enables stable convergence even with incomplete data.
• A novel coherence control technique ensures intermediate solutions remain well-conditioned throughout the iterations.
• The theoretical framework provides a conceptually simple and principled understanding of alternating minimization via subspace iteration.