[Paper Review] Convex Sparse Matrix Factorizations

Francis R. Bach, Julien Mairal+1 more|ArXiv.org|Dec 10, 2008

Sparse and Compressive Sensing TechniquesEngineering18 references105 citations

TL;DR

This paper proposes a convex formulation of sparse dictionary learning by replacing explicit dictionary size constraints with a trace-norm-like rank-reducing regularization, enabling a single global minimum. While the convex approach simplifies optimization and ensures convergence, simulations show it underperforms non-convex methods in high-sparsity and small-dictionary regimes, where non-convex formulations better exploit sparsity for improved denoising.

ABSTRACT

We present a convex formulation of dictionary learning for sparse signal decomposition. Convexity is obtained by replacing the usual explicit upper bound on the dictionary size by a convex rank-reducing term similar to the trace norm. In particular, our formulation introduces an explicit trade-off between size and sparsity of the decomposition of rectangular matrices. Using a large set of synthetic examples, we compare the estimation abilities of the convex and non-convex approaches, showing that while the convex formulation has a single local minimum, this may lead in some cases to performance which is inferior to the local minima of the non-convex formulation.

Motivation & Objective

• To investigate whether convexification of sparse dictionary learning improves estimation performance compared to non-convex alternatives.
• To develop a convex optimization framework that implicitly controls dictionary size via rank regularization instead of explicit bounds.
• To evaluate the trade-off between sparsity of decomposition coefficients and dictionary size using a convex formulation.
• To compare the performance of convex and non-convex formulations in synthetic denoising experiments under varying sparsity and dictionary size.
• To assess whether convexity leads to better generalization or is outperformed by local minima in non-convex formulations under specific conditions.

Proposed method

• Proposes a convex optimization problem that minimizes a loss function over matrix factorizations $X = UV^ op$, with a regularization term promoting low-rank and sparse solutions.
• Introduces a decomposition norm $f_D^M(X) = \min_{UV^\top = X} \sum_{m=1}^M \|u_m\|_C \|v_m\|_R$, which replaces explicit dictionary size constraints with a convex rank-reducing penalty.
• Uses a mixed $\ell^1$-$\ell^2$ regularization framework to jointly promote low-rank structure and sparse coefficient matrices.
• Employs singular value decomposition (SVD) as a baseline for denoising performance comparison.
• Applies a rounding procedure to the convex solution to recover non-convex solutions, enabling comparison with standard non-convex sparse dictionary learning.
• Employs synthetic data generation with random unit-norm dictionary elements and sparse coefficient matrices to simulate realistic signal decomposition tasks.

Experimental results

Research questions

• RQ1 Can sparse dictionary learning be convexified while preserving or improving estimation performance?
• RQ2 Does the convex formulation outperform non-convex methods in terms of mean squared error in matrix denoising tasks?
• RQ3 Under what conditions—specifically regarding sparsity and dictionary size—does the non-convex formulation outperform the convex one?
• RQ4 Is the convex formulation’s single global minimum advantageous in practice, or do local minima of non-convex problems yield better solutions?
• RQ5 Can the convex formulation be rounded to recover high-performing non-convex solutions, and how does this compare to direct non-convex optimization?

Key findings

• In high-sparsity regimes (e.g., $S=2$), the non-convex formulation (NoConv) significantly outperforms the convex formulation (Conv), with relative improvements in mean squared error up to -8.8 and -10.9 in some cases.
• When the ratio $M/P$ is small (e.g., $M=4$, $P=20$), the non-convex method achieves better denoising performance than the convex method, especially in low-dictionary-size settings.
• For medium sparsity ($S=4$), the non-convex method outperforms convex formulations only when $M/P \leq 1$, indicating a threshold effect based on dictionary size relative to signal dimension.
• In low-sparsity regimes ($S=8$), sparsity regularization provides little benefit, and the non-convex method’s performance degrades due to poor local minima, making convex methods more favorable.
• The rounded convex solution (Conv-R) consistently outperforms the unrounded convex solution (Conv), indicating that rounding recovers better local minima of the non-convex problem.
• The convex formulation with proper regularization achieves competitive performance in non-extreme sparsity settings, suggesting it is effective when sparsity is moderate and dictionaries are relatively large.