[Paper Review] Early Stopping for Deep Image Prior

Hengkang Wang, Taihui Li+4 more|arXiv (Cornell University)|Dec 11, 2021

Sparse and Compressive Sensing Techniques43 citations

TL;DR

The paper introduces a simple, effective early-stopping strategy for Deep Image Prior (DIP) and its variants by monitoring the running variance of intermediate reconstructions to detect near-peak performance without ground-truth images. This ES-WMV method works across multiple DIP variants and noise types, improving practicality and efficiency.

ABSTRACT

Deep image prior (DIP) and its variants have showed remarkable potential for solving inverse problems in computer vision, without any extra training data. Practical DIP models are often substantially overparameterized. During the fitting process, these models learn mostly the desired visual content first, and then pick up the potential modeling and observational noise, i.e., overfitting. Thus, the practicality of DIP often depends critically on good early stopping (ES) that captures the transition period. In this regard, the majority of DIP works for vision tasks only demonstrates the potential of the models -- reporting the peak performance against the ground truth, but provides no clue about how to operationally obtain near-peak performance without access to the groundtruth. In this paper, we set to break this practicality barrier of DIP, and propose an efficient ES strategy, which consistently detects near-peak performance across several vision tasks and DIP variants. Based on a simple measure of dispersion of consecutive DIP reconstructions, our ES method not only outpaces the existing ones -- which only work in very narrow domains, but also remains effective when combined with a number of methods that try to mitigate the overfitting. The code is available at https://github.com/sun-umn/Early_Stopping_for_DIP.

Motivation & Objective

• Motivate and address the ELTO (early-learning-then-overfitting) phenomenon in DIP-based inverse problems.
• Develop a practical, ground-truth-free ES criterion that works across DIP variants and noise types/levels.
• Provide a lightweight, robust ES method with minimal hyperparameter sensitivity.
• Demonstrate the ES method as a helper for regularized DIP models and implicit neural representations.
• Offer a comparative analysis against existing ES and NR-IQM-based approaches.

Proposed method

• Parametrize the unknown image with a DIP model and track the reconstruction sequence x^t = G_theta^t(z).
• Define a windowed moving variance (VAR) over recent reconstructions to approximate the ELTO valley without ground-truth.F Implement ES by detecting the valley of the VAR curve with a window size W and a patience P.
• Provide Algorithm 1 for DIP with ES-WMV and an optional memory-efficient variant using exponential moving variance (EMV).
• Justify the approach theoretically under a linearized training regime via neural tangent kernel; show that VAR follows a U-shaped curve with appropriate learning rates.
• Demonstrate robustness to two hyperparameters (window size, patience) and discuss computational overhead being small compared to per-iteration cost.

Experimental results

Research questions

• RQ1 Can an ES criterion based solely on the running variance of intermediate reconstructions reliably identify near-peak DIP performance across tasks?
• RQ2 Is the proposed ES-WMV approach robust and efficient across multiple DIP variants and noise types/levels?
• RQ3 Can ES-WMV serve as a practical helper for regularized DIP methods and implicit neural representations without ground-truth data?
• RQ4 How does ES-WMV compare to existing ES and NR-IQM-based monitoring methods in terms of detection gap and computational cost?
• RQ5 What theoretical insights support the observed VAR valley behavior in DIP training?

Key findings

• The ES-WMV method detects near-peak performance with small PSNR/SSIM gaps across image denoising and blind image deblurring tasks.
• The added per-iteration overhead is small relative to the DIP update step, making ES-WMV efficient.
• ES-WMV remains effective for multiple DIP variants (including DIP-TV, GP-DIP, SIREN) and various noise types/levels.
• NR-IQM baselines often produce larger detection gaps, while ES-WMV provides closer to peak performance without ground-truth.
• ES-WMV improves performance when used as a helper for regularized DIP models (e.g., DIP-TV, GP-DIP) and can aid implicit neural representations like SIREN.
• The method is robust to window size and patience hyperparameters, applying across tasks with minimal tuning.