[Paper Review] Fast and Provable ADMM for Learning with Generative Priors

Fabian Latorre Gómez, Armin Eftekhari+1 more|arXiv (Cornell University)|Jul 7, 2019

Sparse and Compressive Sensing TechniquesEngineering60 references20 citations

TL;DR

This paper proposes a fast, provably convergent linearized ADMM algorithm for optimizing convex objectives with nonconvex generative priors defined by deep neural networks, such as GAN generators. It achieves accelerated convergence by exploiting the geometry of the generator network and efficiently handles non-smooth objectives, outperforming gradient descent in denoising and compressive sensing tasks with theoretical guarantees.

ABSTRACT

In this work, we propose a (linearized) Alternating Direction Method-of-Multipliers (ADMM) algorithm for minimizing a convex function subject to a nonconvex constraint. We focus on the special case where such constraint arises from the specification that a variable should lie in the range of a neural network. This is motivated by recent successful applications of Generative Adversarial Networks (GANs) in tasks like compressive sensing, denoising and robustness against adversarial examples. The derived rates for our algorithm are characterized in terms of certain geometric properties of the generator network, which we show hold for feedforward architectures, under mild assumptions. Unlike gradient descent (GD), it can efficiently handle non-smooth objectives as well as exploit efficient partial minimization procedures, thus being faster in many practical scenarios.

Motivation & Objective

• To address the lack of provably convergent, efficient algorithms for optimization with nonconvex generative priors from GANs.
• To overcome limitations of gradient descent in handling non-smooth objectives like ℓ₁ or ℓ∞-norm denoising.
• To provide convergence guarantees for a linearized ADMM framework applied to nonconvex problems with neural network constraints.
• To establish geometric conditions on the generator network that ensure fast convergence rates.
• To extend theoretical results to statistical learning settings with empirical risk minimization and Rademacher complexity bounds.

Proposed method

• Formulates the optimization problem as minimizing a convex function F(w,z) subject to w = G(z), where G is a differentiable generator network.
• Applies a linearized ADMM framework to handle the nonconvex constraint w = G(z), enabling efficient alternating minimization over w and z.
• Uses proximal mappings for non-smooth regularizers R and H, ensuring computational tractability.
• Derives convergence rates based on geometric properties of G, such as near-isometry and smoothness, under strong convexity of L.
• Performs exact minimization over w via closed-form solution of the augmented Lagrangian, leveraging matrix inversion via SVD when applicable.
• Introduces a primal-dual update scheme with dual variable updates and adaptive step sizes, ensuring stability and convergence.

Experimental results

Research questions

• RQ1 Can a linearized ADMM algorithm achieve provable fast convergence for nonconvex optimization with generative priors?
• RQ2 What geometric properties of the generator network G ensure convergence and fast rates in this framework?
• RQ3 How does the proposed ADMM method compare to gradient descent in handling non-smooth objectives like ℓ₁ and ℓ∞-norms?
• RQ4 Can the algorithm be extended to statistical learning settings with empirical risk, and what generalization error bounds can be derived?
• RQ5 What is the relationship between the proposed ADMM and existing methods like those in Bora et al. (2017)?

Key findings

• The proposed linearized ADMM achieves fast convergence rates under mild geometric assumptions on the generator network, such as near-isometry and smoothness.
• The algorithm provably converges to a solution with convergence rates comparable to gradient descent, but with superior performance on non-smooth objectives.
• For compressive sensing and denoising tasks, the method outperforms standard gradient descent, especially with ℓ₁ and ℓ∞-norms, due to its ability to handle non-smoothness.
• Theoretical convergence is established under strong convexity of L, with extensions to restricted strong convexity for broader statistical learning applications.
• Generalization error bounds are derived using Rademacher complexity, quantifying the number of training samples needed for learning with generative priors.
• The method achieves computational efficiency by enabling exact minimization over w via closed-form solutions and efficient forward/backward passes in deep networks.