[Paper Review] Local Identification of Overcomplete Dictionaries

Karin Schnass|arXiv (Cornell University)|Jan 24, 2014

Sparse and Compressive Sensing Techniques34 references19 citations

TL;DR

This paper establishes the first theoretical conditions under which overcomplete dictionaries with coherence μ can be stably identified from sparse signals using a novel maximisation criterion that generalises K-means. It proves local recovery is possible for sparsity levels up to O(μ⁻²) and signal-to-noise ratios up to O(√d), with finite sample complexity scaling as O(K³dSε̃⁻²), and introduces the ITKM algorithm for efficient local optimisation with O(dKN) per-iteration complexity.

ABSTRACT

This paper presents the first theoretical results showing that stable identification of overcomplete $μ$-coherent dictionaries $Φ\in \mathbb{R}^{d imes K}$ is locally possible from training signals with sparsity levels $S$ up to the order $O(μ^{-2})$ and signal to noise ratios up to $O(\sqrt{d})$. In particular the dictionary is recoverable as the local maximum of a new maximisation criterion that generalises the K-means criterion. For this maximisation criterion results for asymptotic exact recovery for sparsity levels up to $O(μ^{-1})$ and stable recovery for sparsity levels up to $O(μ^{-2})$ as well as signal to noise ratios up to $O(\sqrt{d})$ are provided. These asymptotic results translate to finite sample size recovery results with high probability as long as the sample size $N$ scales as $O(K^3dS ilde \varepsilon^{-2})$, where the recovery precision $ ilde \varepsilon$ can go down to the asymptotically achievable precision. Further, to actually find the local maxima of the new criterion, a very simple Iterative Thresholding and K (signed) Means algorithm (ITKM), which has complexity $O(dKN)$ in each iteration, is presented and its local efficiency is demonstrated in several experiments.

Motivation & Objective

• To establish theoretical conditions for stable local identification of overcomplete dictionaries from finite training signals.
• To close the gap between existing theoretical bounds (O(μ⁻¹)) and empirical performance (O(μ⁻²)) in dictionary learning.
• To propose a new optimisation criterion generalising K-means that enables local recovery of dictionaries under higher sparsity levels.
• To derive finite sample size recovery guarantees with explicit sample complexity scaling.
• To introduce and validate the ITKM algorithm for efficient local optimisation of the new criterion.

Proposed method

• Proposes a new maximisation criterion that generalises the K-means objective to enable stable dictionary recovery from sparse signals.
• Derives asymptotic recovery guarantees showing exact recovery up to O(μ⁻¹) sparsity and stable recovery up to O(μ⁻²) sparsity.
• Establishes finite sample size recovery bounds by controlling deviation of empirical criteria from their expectations using concentration inequalities.
• Introduces the Iterative Thresholding and K (signed) Means (ITKM) algorithm with O(dKN) complexity per iteration, combining thresholding and signed K-means updates.
• Uses a probabilistic analysis of the criterion's landscape to show that the true dictionary is a local maximum under the new criterion.
• Derives sample complexity bounds by balancing estimation error and deviation probabilities, leading to N = O(K³dSε̃⁻²) for precision ε̃.

Experimental results

Research questions

• RQ1 Can overcomplete dictionaries with coherence μ be stably identified from sparse signals beyond the O(μ⁻¹) sparsity threshold?
• RQ2 Is there a principled optimisation criterion that generalises K-means and enables local dictionary recovery under higher sparsity levels?
• RQ3 What is the required sample size for finite-sample recovery with high probability under the new criterion?
• RQ4 Can a simple iterative algorithm like ITKM achieve local convergence to the true dictionary?
• RQ5 How does the signal-to-noise ratio affect the recovery threshold in the proposed framework?

Key findings

• Local identification of overcomplete μ-coherent dictionaries is possible for sparsity levels up to O(μ⁻²), significantly exceeding prior theoretical limits of O(μ⁻¹).
• The proposed maximisation criterion ensures asymptotic exact recovery for sparsity up to O(μ⁻¹) and stable recovery up to O(μ⁻²).
• Finite sample recovery is guaranteed with high probability when the sample size scales as O(K³dSε̃⁻²), where ε̃ is the target recovery precision.
• The ITKM algorithm achieves local convergence to the true dictionary with O(dKN) complexity per iteration, demonstrating practical feasibility.
• Stable recovery is possible even under signal-to-noise ratios up to O(√d), extending the regime of applicability.
• Theoretical bounds on sample complexity are derived by controlling deviation of empirical criteria from their expectations using concentration of measure.