[Paper Review] Compressed Sensing using Generative Models

Ashish Bora, Ajil Jalal+2 more|arXiv (Cornell University)|Mar 9, 2017

Sparse and Compressive Sensing Techniques30 references291 citations

TL;DR

The paper replaces sparsity with generative priors for compressed sensing, showing that random Gaussian measurements of size O(kd log n) suffice for accurate recovery via gradient descent in the generator’s latent space, often with 5–10x fewer measurements than Lasso.

ABSTRACT

The goal of compressed sensing is to estimate a vector from an underdetermined system of noisy linear measurements, by making use of prior knowledge on the structure of vectors in the relevant domain. For almost all results in this literature, the structure is represented by sparsity in a well-chosen basis. We show how to achieve guarantees similar to standard compressed sensing but without employing sparsity at all. Instead, we suppose that vectors lie near the range of a generative model $G: \mathbb{R}^k o \mathbb{R}^n$. Our main theorem is that, if $G$ is $L$-Lipschitz, then roughly $O(k \log L)$ random Gaussian measurements suffice for an $\ell_2/\ell_2$ recovery guarantee. We demonstrate our results using generative models from published variational autoencoder and generative adversarial networks. Our method can use $5$-$10$x fewer measurements than Lasso for the same accuracy.

Motivation & Objective

• Motivate alternative structure priors for compressed sensing beyond sparsity.
• Formalize a theoretical framework (S/REC) for the range of a generative model.
• Show that Gaussian measurement matrices satisfy S/REC for broad generator classes.
• Provide recovery guarantees when optimizing in latent space using gradient descent.
• Demonstrate practical performance on real datasets with VAEs and GANs.

Proposed method

• Formulate an optimization where z in R^k minimizes ||A G(z) - y||_2^2 and recover x̂ = G(ẑ).
• Introduce a regularizer L(z) to encourage generator-preferred regions, e.g., L(z)=λ||z||^2.
• Establish the Set-Restricted Eigenvalue Condition (S/REC) as a generalization of REC for S = range(G).
• Prove that random Gaussian A satisfies S/REC(G(B^k(r)), 1−α, δ) under mild bounds on m.
• Derive error bounds linking reconstruction error to the best approximation within the generator’s range and measurement/optimization errors.
• Provide Lipschitz (L) and Lipschitz-network (d-layer) guarantees giving m = O(k log L) or m = O(kd log n).

Experimental results

Research questions

• RQ1 How many Gaussian measurements m are needed to reliably recover x* via G(z) from y=Ax*+η?
• RQ2 Can gradient descent in the generator’s latent space recover x* with provable guarantees?
• RQ3 How does recovery error relate to the best approximation within the generator’s range and measurement/optimization errors?
• RQ4 Do results extend from ReLU nets to arbitrary L-Lipschitz generators?
• RQ5 What are the practical performance implications when using VAEs and GANs on real data (MNIST, CelebA) compared to Lasso?

Key findings

• Gaussian matrices satisfy S/REC for generator ranges with high probability, enabling recovery guarantees.
• For d-layer neural networks (VAEs/GANs), m = O(kd log n) measurements suffice for good reconstruction with high probability.
• The latent-space optimization recovers x̂ close to x* within the best generator-range approximation, plus terms proportional to noise and optimization error.
• Empirical results show the method uses 5–10x fewer measurements than Lasso to achieve similar accuracy in some regimes.
• On MNIST and CelebA, reconstructions from few Gaussian measurements are competitive, with representation error identified as a major component of total error.
• Super-resolution experiments demonstrate sharp reconstructions when measurements align with the generator’s range constraints.