[Paper Review] Complete Dictionary Recovery over the Sphere

Ju Sun, Qing Qu+1 more|arXiv (Cornell University)|Apr 26, 2015

Sparse and Compressive Sensing Techniques11 references40 citations

TL;DR

This paper presents the first provably efficient algorithm for complete dictionary recovery from sparse measurements, leveraging a Riemannian trust region method on the sphere to avoid spurious local minima. It establishes that exact recovery of the full invertible dictionary is possible when each signal has O(n) nonzeros, significantly improving over prior methods requiring sparsity levels of O(n^{1−δ}).

ABSTRACT

We consider the problem of recovering a complete (i.e., square and invertible) matrix $\mathbf A_0$, from $\mathbf Y \in \mathbb R^{n imes p}$ with $\mathbf Y = \mathbf A_0 \mathbf X_0$, provided $\mathbf X_0$ is sufficiently sparse. This recovery problem is central to the theoretical understanding of dictionary learning, which seeks a sparse representation for a collection of input signals, and finds numerous applications in modern signal processing and machine learning. We give the first efficient algorithm that provably recovers $\mathbf A_0$ when $\mathbf X_0$ has $O(n)$ nonzeros per column, under suitable probability model for $\mathbf X_0$. In contrast, prior results based on efficient algorithms provide recovery guarantees when $\mathbf X_0$ has only $O(n^{1-δ})$ nonzeros per column for any constant $δ\in (0, 1)$. Our algorithmic pipeline centers around solving a certain nonconvex optimization problem with a spherical constraint, and hence is naturally phrased in the language of manifold optimization. To show this apparently hard problem is tractable, we first provide a geometric characterization of the high-dimensional objective landscape, which shows that with high probability there are no "spurious" local minima. This particular geometric structure allows us to design a Riemannian trust region algorithm over the sphere that provably converges to one local minimizer with an arbitrary initialization, despite the presence of saddle points. The geometric approach we develop here may also shed light on other problems arising from nonconvex recovery of structured signals.

Motivation & Objective

• Address the fundamental challenge of recovering a complete, invertible dictionary matrix from sparse linear measurements.
• Overcome the limitation of prior efficient algorithms that only guarantee recovery under suboptimal sparsity levels (O(n^{1−δ})).
• Provide a theoretical guarantee for exact dictionary recovery when the sparsity per column scales linearly with the dimension n.
• Develop a geometric framework to analyze the nonconvex optimization landscape of dictionary learning over the sphere.
• Design a provably convergent algorithm for dictionary recovery that works with arbitrary initialization despite the presence of saddle points.

Proposed method

• Formulate dictionary recovery as a nonconvex optimization problem with a spherical constraint on the dictionary atoms.
• Use Riemannian trust region optimization to navigate the manifold of orthogonal matrices, ensuring convergence to a local minimizer.
• Establish that the optimization landscape contains no spurious local minima with high probability under a random sparse model.
• Leverage geometric analysis to prove that all critical points are either global minima or saddle points, enabling global convergence.
• Introduce a novel manifold-based algorithm that avoids local minima and converges to a solution with arbitrary initialization.
• Prove convergence to a global minimizer using Riemannian optimization theory, even in the presence of saddle points.

Experimental results

Research questions

• RQ1 Can a provably efficient algorithm recover a complete dictionary when the number of nonzeros per signal scales linearly with dimension?
• RQ2 What is the geometric structure of the nonconvex optimization landscape in dictionary learning over the sphere?
• RQ3 Are there spurious local minima in the dictionary recovery problem under the spherical constraint?
• RQ4 Can a Riemannian optimization method converge globally to a solution with arbitrary initialization?
• RQ5 What conditions on the sparsity level and signal model ensure exact recovery of the true dictionary?

Key findings

• The proposed algorithm provably recovers the true dictionary with high probability when each column of the sparse coefficient matrix has O(n) nonzeros.
• The nonconvex optimization landscape over the sphere contains no spurious local minima with high probability, enabling global convergence.
• The Riemannian trust region method converges to a global minimizer from any initialization, despite the presence of saddle points.
• The theoretical guarantees improve upon prior work by extending the allowable sparsity level from O(n^{1−δ}) to O(n), a significant improvement.
• The geometric framework developed in this work may be applicable to other nonconvex structured signal recovery problems.
• The original long paper was split into two follow-up papers (arXiv:1511.03607 and arXiv:1511.04777), indicating the depth and significance of the results.