[Paper Review] Recovery of Sparse Signals from a Mixture of Linear Samples

Arya Mazumdar, Soumyabrata Pal|arXiv (Cornell University)|Jun 29, 2020

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes an efficient algorithm for recovering two unknown sparse linear models from a mixture of linear queries, where each query returns a label from one of the two models without identifying which. By leveraging compressed sensing and basis pursuit with carefully designed queries, the method achieves optimal query complexity without restrictive assumptions on sparsity or model parameters, significantly improving prior work in sample efficiency and generality.

ABSTRACT

Mixture of linear regressions is a popular learning theoretic model that is used widely to represent heterogeneous data. In the simplest form, this model assumes that the labels are generated from either of two different linear models and mixed together. Recent works of Yin et al. and Krishnamurthy et al., 2019, focus on an experimental design setting of model recovery for this problem. It is assumed that the features can be designed and queried with to obtain their label. When queried, an oracle randomly selects one of the two different sparse linear models and generates a label accordingly. How many such oracle queries are needed to recover both of the models simultaneously? This question can also be thought of as a generalization of the well-known compressed sensing problem (Candès and Tao, 2005, Donoho, 2006). In this work, we address this query complexity problem and provide efficient algorithms that improves on the previously best known results.

Motivation & Objective

• Address the query complexity problem in learning a mixture of two sparse linear models when features can be designed and queried.
• Overcome the limitations of prior work that required restrictive assumptions such as discrete-valued coefficients or exact sparsity.
• Provide a general-purpose recovery algorithm that works for arbitrary sparse vectors without prior knowledge of coefficient structure.
• Achieve high-probability recovery of both models with minimal queries, even when the models are close in parameter space.
• Establish query complexity bounds that scale favorably with sparsity, signal-to-noise ratio, and desired recovery precision.

Proposed method

• Design a two-stage query strategy: first, use random projections to separate the two models based on their distinct linear responses.
• Leverage the restricted isometric property (RIP) of the measurement matrix to ensure stable recovery of sparse signals.
• Apply basis pursuit via convex optimization to estimate each model from grouped samples, minimizing the L1 norm under a data-fidelity constraint.
• Use Chernoff bounds to ensure high-probability concentration of informative queries, where the difference between model responses is detectable above noise.
• Define the noise factor NF = γ/σ and signal-to-noise ratio (SNR) to characterize query efficiency and derive sample complexity in terms of these parameters.
• Introduce a permutation-invariant recovery guarantee, where the estimated models are correctly matched to the true ones up to labeling.

Experimental results

Research questions

• RQ1 What is the minimum number of queries required to recover two unknown sparse linear models from a mixture of their responses?
• RQ2 How can we design queries such that the resulting samples allow separation and recovery of both models without prior knowledge of their structure?
• RQ3 Can we eliminate restrictive assumptions—such as discrete-valued coefficients or exact sparsity—while maintaining low query complexity?
• RQ4 How does the query complexity scale with sparsity k, dimension n, noise level σ, and desired recovery precision γ?
• RQ5 What is the impact of the difference between the two models (‖β¹ − β²‖₂) on the required number of queries?

Key findings

• The proposed algorithm achieves a query complexity of O(k log n log k ⌈log k / log(√SNR / NF)⌉ ⌈1/(NF⁴√SNR) + 1/NF²⌉), which improves upon prior work by removing restrictive assumptions.
• The method works for general sparse vectors without requiring exact sparsity or discrete coefficient values, making it applicable to real-world data.
• For γ < ‖β¹ − β²‖₂ / 2, the algorithm recovers both models with high probability and error bounded by O(γ) plus a term related to the best k-sparse approximation.
• When γ = Ω(‖β¹ − β²‖₂), the problem reduces to single-model recovery, and the query complexity simplifies to O(k log n ⌈log k⌉), matching standard compressed sensing bounds.
• The analysis shows that the number of informative queries concentrates tightly around its expectation, ensuring reliable recovery with high probability.
• The sample complexity depends only on k, n, SNR, and the precision parameter γ, with no exponential dependence on 1/ε, unlike prior approaches.