[Paper Review] Intermediate Layer Optimization for Inverse Problems using Deep Generative Models

Giannis Daras, Joseph Dean+2 more|arXiv (Cornell University)|Feb 15, 2021

Sparse and Compressive Sensing TechniquesEngineering59 references26 citations

TL;DR

ILO progressively optimizes intermediate generator layers to solve inverse problems with deep generative priors, improving reconstruction quality over state-of-the-art methods. It provides theoretical error bounds and broad empirical wins across inpainting, denoising, super-resolution, and compressed sensing.

ABSTRACT

We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small $l_1$ ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.

Motivation & Objective

• Motivate solving inverse problems using pre-trained deep generators as priors.
• Extend existing generator inversion by progressively optimizing intermediate layers to increase expressive power.
• Provide theoretical guarantees showing improved error bounds under constrained latent searches.
• Demonstrate empirical superiority of ILO over baselines on diverse tasks (inpainting, denoising, super-resolution, CS).
• Open-source code to foster reproducibility and further research.

Proposed method

• Decompose a pre-trained generator G into G = G2 ∘ G1 and optimize latent codes at the input of G1 and then progressively at the input of G2.
• At each round, search for latent codes within an l1-ball around the manifold induced by the previous layer to control realism.
• Perform projection back to the generator range by solving a latent-minding step to minimize ||G1(z^k) − z̃^p|| and re-project to the generator range.
• Use gradient-based optimization with differentiable forward operators; apply PGD with customized l1-ball projections.
• Theoretical analysis yields a sample complexity bound showing improved recovery guarantees when expanding the range via a small l1-ball (extended range).
• Adapt StyleGAN-2 specifics by using the mapping network (z to w) and exploring multiple intermediate layers with regularization and loss-function choices (MSE, LPIPS, or their combination).

Experimental results

Research questions

• RQ1 Can progressive optimization over intermediate generator layers improve inversion quality for a broad class of inverse problems?
• RQ2 How does constraining latent-code searches to a small l1-ball around previous layers affect sample complexity and reconstruction error?
• RQ3 Do intermediate-layer optimizations extend applicability beyond super-resolution to tasks like inpainting, denoising, and compressed sensing with deep priors?
• RQ4 What are practical considerations and losses when integrating ILO with StyleGAN-2 for high-resolution images?
• RQ5 Is it possible to guide generation with differentiable forward operators such as classifiers to produce targeted or out-of-distribution images while maintaining realism?

Key findings

• ILO consistently outperforms state-of-the-art methods based on CSGM and PULSE across inpainting, denoising, super-resolution, and compressed sensing experiments.
• Theoretical results show improved error bounds when searches are restricted to a small l1-ball around the manifold of the previous layer, under sufficient measurements.
• Empirical results on CelebA-HQ indicate 2× reconstruction improvement over baselines at 5–100% observed pixels for inpainting.
• ILO achieves higher PSNR/quality in denoising and superior reconstruction for various downsampling factors in super-resolution compared to PULSE and CSGM-based variants.
• The approach enables high-resolution image reconstruction with structured measurements (partial circulant matrices) and supports adversarially robust classifier-guided generation.
• Code is open-sourced to promote reproducibility and further research.