[Paper Review] Working Locally Thinking Globally - Part I: Theoretical Guarantees for Convolutional Sparse Coding

Vardan Papyan, Jeremias Sulam+1 more|arXiv (Cornell University)|Jul 7, 2016

Regional Development and Policy4 citations

TL;DR

This paper establishes the first theoretical guarantees for convolutional sparse coding by introducing novel concepts like shifted mutual coherence and stripe coherence to ensure uniqueness and success of pursuit algorithms. It proves that Orthogonal Matching Pursuit can exactly recover the sparse code under conditions on the dictionary's coherence structure, providing a rigorous foundation for global sparse modeling with local processing.

ABSTRACT

The celebrated sparse representation model has led to remarkable results in various signal processing tasks in the last decade. However, despite its initial purpose of serving as a global prior for entire signals, it has been commonly used for modeling low dimensional patches due to the computational constraints it entails when deployed with learned dictionaries. A way around this problem has been proposed recently, adopting a convolutional sparse representation model. This approach assumes that the global dictionary is a concatenation of banded Circulant matrices. Although several works have presented algorithmic solutions to the global pursuit problem under this new model, very few truly-effective guarantees are known for the success of such methods. In the first of this two-part work, we address the theoretical aspects of the sparse convolutional model, providing the first meaningful answers to corresponding questions of uniqueness of solutions and success of pursuit algorithms. To this end, we generalize mathematical quantities, such as the $\ell_0$ norm, the mutual coherence and the Spark, to their counterparts in the convolutional setting, which intrinsically capture local measures of the global model. In a companion paper, we extend the analysis to a noisy regime, addressing the stability of the sparsest solutions and pursuit algorithms, and demonstrate practical approaches for solving the global pursuit problem via simple local processing.

Motivation & Objective

• To establish theoretical conditions ensuring uniqueness of sparse solutions in convolutional sparse coding.
• To provide provable guarantees for the success of global pursuit algorithms like Orthogonal Matching Pursuit under the convolutional model.
• To generalize classical sparse coding concepts—such as mutual coherence and Spark—to the convolutional setting with local structure.
• To lay the theoretical groundwork for global signal modeling using local processing by analyzing the coherence properties of banded circulant dictionaries.

Proposed method

• Introduce the concept of shifted mutual coherence μs to capture local correlations between atoms in the convolutional dictionary.
• Define stripe coherence ζk as the sum of shifted mutual coherences across a support region, enabling global recovery analysis.
• Prove that if maxk ζk < ½(1 + μ0), then Orthogonal Matching Pursuit successfully recovers the true sparse support in each iteration.
• Use a decomposition of the residual and inner product bounds to show that the algorithm selects atoms from the true support at each step.
• Establish a hierarchy of recovery conditions, showing that the stripe coherence condition is stronger than the ℓ0,∞-norm condition.
• Leverage the banded circulant structure of the global dictionary to enable local processing while maintaining global reconstruction guarantees.

Experimental results

Research questions

• RQ1 Under what conditions is the sparse solution in convolutional sparse coding unique?
• RQ2 Can global pursuit algorithms like Orthogonal Matching Pursuit be guaranteed to recover the true sparse code in the convolutional model?
• RQ3 How do classical sparse coding concepts like mutual coherence and Spark generalize to the convolutional setting?
• RQ4 What coherence-based condition ensures the success of Orthogonal Matching Pursuit in recovering the true support?
• RQ5 Is the proposed coherence condition stronger than existing recovery conditions based on ℓ0,∞-norm?

Key findings

• The paper proves that if the maximum stripe coherence satisfies maxk ζk < ½(1 + μ0), then Orthogonal Matching Pursuit exactly recovers the true sparse code in ‖Γ‖0 iterations.
• The proposed stripe coherence condition is strictly stronger than the classical ℓ0,∞-norm condition, meaning it guarantees recovery under broader or more realistic settings.
• The shifted mutual coherence μs captures local dependencies between atoms in the convolutional dictionary, enabling a more accurate analysis than global mutual coherence.
• The analysis shows that μ0 (the zero-shift coherence) is typically the largest, justifying its use as a baseline in the recovery condition.
• The theoretical framework enables global modeling with local processing by ensuring that local pursuit operations can recover the global sparse representation.
• The results provide the first rigorous theoretical foundation for convolutional sparse coding, addressing long-standing gaps in understanding its success and stability.