[Paper Review] Sparse Solutions to Nonnegative Linear Systems and Applications

Aditya Bhaskara, Ananda Theertha Suresh+1 more|arXiv (Cornell University)|Jan 7, 2015

Machine Learning and Algorithms36 references3 citations

TL;DR

This paper presents a combinatorial algorithm for finding sparse approximate solutions to nonnegative linear systems without requiring matrix structure assumptions. It guarantees an $ O(k/ heta^3) $-sparse solution with $ \ell_1 $ error $ \leq \theta\|b\|_1 $, enabling efficient PAC learning of $ k $-component axis-aligned Gaussian mixtures in $ d $ dimensions with $ O(kd/\theta^3)^d $ complexity when $ d $ is constant.

ABSTRACT

We give an efficient algorithm for finding sparse approximate solutions to linear systems of equations with nonnegative coefficients. Unlike most known results for sparse recovery, we do not require {\em any} assumption on the matrix other than non-negativity. Our algorithm is combinatorial in nature, inspired by techniques for the set cover problem, as well as the multiplicative weight update method. We then present a natural application to learning mixture models in the PAC framework. For learning a mixture of $k$ axis-aligned Gaussians in $d$ dimensions, we give an algorithm that outputs a mixture of $O(k/ε^3)$ Gaussians that is $ε$-close in statistical distance to the true distribution, without any separation assumptions. The time and sample complexity is roughly $O(kd/ε^3)^{d}$. This is polynomial when $d$ is constant -- precisely the regime in which known methods fail to identify the components efficiently. Given that non-negativity is a natural assumption, we believe that our result may find use in other settings in which we wish to approximately explain data using a small number of a (large) candidate set of components.

Motivation & Objective

• To develop an efficient algorithm for sparse approximate solutions to nonnegative linear systems without structural assumptions on the matrix.
• To address the challenge of sparse recovery in $ \ell_1 $-norm error, which is poorly handled by existing convex relaxation or greedy methods.
• To enable PAC learning of mixture models—specifically $ k $-component axis-aligned Gaussians—without separation assumptions or reliance on matrix properties.
• To provide a combinatorial, multiplicative weight-inspired method that avoids convex optimization and eigenvalue dependencies.
• To achieve polynomial-time learning in low-dimensional regimes where prior methods fail due to exponential complexity.

Proposed method

• The algorithm uses a multiplicative weight update framework inspired by set cover techniques, iteratively adjusting weights to focus on poorly explained components.
• It maintains a nonnegative solution vector and updates it using a greedy selection rule based on residual error in the $ \ell_1 $-norm.
• The method avoids convex relaxation by relying on combinatorial updates and does not require restricted isometry or incoherence conditions.
• It leverages a probabilistic argument using the Paley-Zygmund inequality to lower-bound the probability that a sparse vector fails to approximate the target.
• An $ \epsilon' $-net over sparse vectors is constructed to allow a union bound over all candidate solutions, ensuring global approximation guarantees.
• The analysis uses concentration bounds and moment estimates to show that with high probability, no $ Ck $-sparse vector with $ \|x\|_1 \in [3k/4, 5k/4] $ can achieve small $ \ell_1 $ error, implying the existence of a good sparse solution.

Experimental results

Research questions

• RQ1 Can sparse approximate solutions be found efficiently for nonnegative linear systems without structural assumptions on the matrix?
• RQ2 Is it possible to achieve $ \ell_1 $-norm error guarantees in sparse recovery without relying on restricted isometry or convex relaxation?
• RQ3 Can this method be applied to PAC learning of mixture models in low-dimensional settings where known methods fail?
• RQ4 What is the trade-off between sparsity and $ \ell_1 $-error in nonnegative systems when no exact sparse solution exists?
• RQ5 Can a combinatorial, non-convex method outperform convex relaxation in terms of sparsity-accuracy trade-offs for nonnegative systems?

Key findings

• The algorithm produces a solution that is $ O(k/\epsilon^3) $-sparse and satisfies $ \|Ax_{\text{alg}} - b\|_1 \leq \epsilon\|b\|_1 $, even without matrix structure assumptions.
• For learning $ k $-component axis-aligned Gaussians in $ d $ dimensions, the method outputs a mixture of $ O(k/\epsilon^3) $ components that is $ \epsilon $-close in statistical distance.
• The time and sample complexity is $ O((kd/\epsilon^3)^d) $, which is polynomial when $ d $ is constant—resolving a key limitation of prior methods.
• The method avoids dependence on the largest eigenvalue of $ A^\dagger $, unlike Natarajan’s approach, and does not require RIP-style assumptions.
• The analysis shows that with high probability, no $ Ck $-sparse vector with $ \|x\|_1 \in [3k/4, 5k/4] $ can achieve $ \ell_1 $ error less than $ \Omega(1/\sqrt{C}) $, implying a lower bound on approximation quality.
• The result demonstrates that non-negativity alone enables strong sparse recovery guarantees, suggesting broad applicability in data modeling with candidate components.