[Paper Review] Mathematics of Sparsity and Entropy: Axioms, Core Functions and Sparse Recovery

Giancarlo Pastor, Inmaculada Mora-Jiménez+2 more|arXiv (Cornell University)|Jan 21, 2015

Sparse and Compressive Sensing Techniques41 references21 citations

TL;DR

This paper introduces a unified axiomatic framework that jointly characterizes sparsity and entropy through core functions, generalizing key measures like the Hoyer measure, Gini index, Rényi and Tsallis entropies. The core sparsity function provides theoretical justification for ℓ₁-minimization in compressed sensing, demonstrating its foundational role in sparse recovery.

ABSTRACT

Sparsity and entropy are pillar notions of modern theories in signal processing and information theory. However, there is no clear consensus among scientists on the characterization of these notions. Previous efforts have contributed to understand individually sparsity or entropy from specific research interests. This paper proposes a mathematical formalism, a joint axiomatic characterization, which contributes to comprehend (the beauty of) sparsity and entropy. The paper gathers and introduces inherent and first principles criteria as axioms and attributes that jointly characterize sparsity and entropy. The proposed set of axioms is constructive and allows to derive simple or \emph{core functions} and further generalizations. Core sparsity generalizes the Hoyer measure, Gini index and $pq$-means. Core entropy generalizes the Rényi entropy and Tsallis entropy, both of which generalize Shannon entropy. Finally, core functions are successfully applied to compressed sensing and to minimum entropy given sample moments. More importantly, the (simplest) core sparsity adds theoretical support to the $\ell_1$-minimization approach in compressed sensing.

Motivation & Objective

• To resolve the lack of consensus on formal characterizations of sparsity and entropy in signal processing and information theory.
• To establish a joint axiomatic foundation that unifies individual treatments of sparsity and entropy from first principles.
• To derive constructive core functions that generalize existing sparsity and entropy measures.
• To apply the core functions to practical problems such as compressed sensing and minimum entropy estimation under moment constraints.

Proposed method

• Proposes a set of axioms based on inherent and first-principles criteria that jointly characterize sparsity and entropy.
• Derives 'core functions' as minimal, constructive representations from the axiomatic system, serving as foundational measures.
• Core sparsity generalizes the Hoyer measure, Gini index, and pq-means; core entropy generalizes Rényi and Tsallis entropies.
• Applies core sparsity to compressed sensing, showing theoretical support for ℓ₁-minimization as a recovery method.
• Uses core functions to solve the problem of minimum entropy given sample moments, demonstrating practical utility.

Experimental results

Research questions

• RQ1 How can sparsity and entropy be formally and jointly characterized using a unified axiomatic framework?
• RQ2 What are the minimal, fundamental functions (core functions) that generalize existing sparsity and entropy measures?
• RQ3 How do the derived core functions support or justify standard methods like ℓ₁-minimization in compressed sensing?
• RQ4 Can the core functions be applied to constrained optimization problems such as minimum entropy under moment constraints?
• RQ5 What is the theoretical relationship between the proposed core functions and established measures like Shannon, Rényi, and Tsallis entropy?

Key findings

• The proposed axiomatic system successfully unifies sparsity and entropy through a set of constructive, first-principles axioms.
• Core sparsity generalizes the Hoyer measure, Gini index, and pq-means, providing a broader theoretical basis for sparsity quantification.
• Core entropy generalizes Rényi and Tsallis entropies, which in turn generalize Shannon entropy, extending the scope of entropy measures.
• The simplest core sparsity function provides theoretical justification for the ℓ₁-minimization approach in compressed sensing.
• The core functions are successfully applied to the problem of minimum entropy given sample moments, demonstrating their practical relevance.
• The framework establishes a foundational link between axiomatic principles and widely used signal processing techniques, enhancing theoretical understanding.