[Paper Review] Sparse and spurious: dictionary learning with noise and outliers

Rémi Gribonval, Rodolphe Jenatton+1 more|arXiv (Cornell University)|Jul 19, 2014

Sparse and Compressive Sensing Techniques42 references5 citations

TL;DR

This paper establishes the existence of a local minimum in sparse dictionary learning under noisy and outlier-ridden conditions, proving that with high probability, the true reference dictionary is recovered near a local optimum. It extends prior work by analyzing overcomplete, noisy, and outlier-affected signals using non-asymptotic probabilistic analysis of coherence and sparsity scaling.

ABSTRACT

A popular approach within the signal processing and machine learning communities consists in modelling signals as sparse linear combinations of atoms selected from a learned dictionary. While this paradigm has led to numerous empirical successes in various fields ranging from image to audio processing, there have only been a few theoretical arguments supporting these evidences. In particular, sparse coding, or sparse dictionary learning, relies on a non-convex procedure whose local minima have not been fully analyzed yet. In this paper, we consider a probabilistic model of sparse signals, and show that, with high probability, sparse coding admits a local minimum around the reference dictionary generating the signals. Our study takes into account the case of over-complete dictionaries, noisy signals, and possible outliers, thus extending previous work limited to noiseless settings and/or under-complete dictionaries. The analysis we conduct is non-asymptotic and makes it possible to understand how the key quantities of the problem, such as the coherence or the level of noise, can scale with respect to the dimension of the signals, the number of atoms, the sparsity and the number of observations.

Motivation & Objective

• To theoretically justify the empirical success of sparse dictionary learning in the presence of noise and outliers.
• To analyze the existence of a local minimum near the true dictionary in non-convex sparse coding optimization.
• To extend prior identifiability results beyond noiseless and undercomplete settings to realistic signal conditions.
• To quantify how key parameters—such as coherence, noise level, sparsity, and signal dimension—affect recovery guarantees.
• To provide non-asymptotic bounds on the stability and robustness of dictionary learning under probabilistic signal models.

Proposed method

• Uses a probabilistic model of sparse signals with bounded, non-zero coefficients and additive noise to model realistic signal generation.
• Analyzes the optimization landscape of the sparse coding cost function via the expected difference between the true and estimated reconstruction error.
• Applies a cumulative coherence assumption and restricted isometry property (RIP) to control the behavior of the optimization objective near the true dictionary.
• Employs exact recovery arguments (Proposition 3) to show that the proxy cost function approximates the true objective in a neighborhood of the reference dictionary.
• Leverages convex duality and high-probability concentration bounds to replace almost-sure results with high-probability guarantees, improving robustness.
• Introduces a framework that allows replacing worst-case bounds with expected-value-based bounds, reducing conservatism in theoretical guarantees.

Experimental results

Research questions

• RQ1 Under what conditions does sparse dictionary learning admit a local minimum near the true reference dictionary in the presence of noise?
• RQ2 How does the coherence of the dictionary and the sparsity level affect the stability of the optimization landscape?
• RQ3 Can theoretical guarantees for dictionary recovery be extended to overcomplete dictionaries under noisy and outlier-contaminated signals?
• RQ4 What is the relationship between the radius of the neighborhood around the true dictionary where local minima exist and the signal-to-noise ratio or sparsity?
• RQ5 Can high-probability recovery results replace almost-sure results to improve the robustness and practical relevance of theoretical bounds?

Key findings

• With high probability, a local minimum of the sparse coding objective exists in a neighborhood of the true dictionary, even under noise and outliers.
• The radius of this neighborhood is bounded by $ O(1) $ in Frobenius norm, which is tight due to symmetry in dictionary atoms.
• The analysis shows that the expected difference between the true and proxy cost functions is positive in a ball of radius $ r = O(1) $, ensuring local optimality.
• The method allows replacing conservative worst-case bounds (e.g., $ M_{oldsymbol{eta}} $) with expected-value-based bounds, improving practical relevance.
• Theoretical guarantees can be extended to very overcomplete dictionaries (beyond $ p rianglelesssim m^2 $) by relaxing assumptions on the range of coefficients.
• High-probability recovery results can be derived using concentration inequalities, replacing almost-sure assumptions and improving robustness to model misspecification.