[Paper Review] NAS-DIP: Learning Deep Image Prior with Neural Architecture Search

Yunchun Chen, Chen Gao+2 more|arXiv (Cornell University)|Aug 26, 2020

Advanced Image Processing Techniques85 references4 citations

TL;DR

This paper proposes NAS-DIP, a method that uses neural architecture search (NAS) to automatically discover optimal upsampling cells and cross-scale residual connection patterns in U-Net-based architectures for learning image priors. By leveraging reinforcement learning with a search space over upsampling operations and skip connections, NAS-DIP achieves state-of-the-art performance among learning-free methods and competitive results with learning-based models across image restoration and synthesis tasks.

ABSTRACT

Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior for solving various inverse image restoration tasks. Instead of using hand-designed architectures, we propose to search for neural architectures that capture stronger image priors. Building upon a generic U-Net architecture, our core contribution lies in designing new search spaces for (1) an upsampling cell and (2) a pattern of cross-scale residual connections. We search for an improved network by leveraging an existing neural architecture search algorithm (using reinforcement learning with a recurrent neural network controller). We validate the effectiveness of our method via a wide variety of applications, including image restoration, dehazing, image-to-image translation, and matrix factorization. Extensive experimental results show that our algorithm performs favorably against state-of-the-art learning-free approaches and reaches competitive performance with existing learning-based methods in some cases.

Motivation & Objective

• To improve learning-free image restoration by discovering better neural architectures for structured image priors.
• To address the under-explored design of decoder upsampling cells and fixed skip connections in existing deep image prior methods.
• To develop a search space for both upsampling operations and cross-level residual connections in encoder-decoder networks.
• To validate the effectiveness of NAS-DIP across diverse image restoration and synthesis tasks without large-scale labeled datasets.
• To demonstrate that NAS can discover architectures that outperform hand-designed models like U-Net and DIP in learning-free settings.

Proposed method

• Design a search space for upsampling cells by decomposing the operation into spatial resolution change (e.g., bilinear, nearest neighbor) and feature transformation (e.g., 2D transposed convolution).
• Develop a search space for cross-scale residual connections that allows flexible, learned patterns of feature fusion between encoder and decoder layers.
• Use a reinforcement learning-based NAS framework with a recurrent neural network controller to explore the architecture space, using PSNR as the reward signal.
• Train the controller to discover optimal architectures for specific tasks, such as super-resolution, denoising, inpainting, and dehazing, without requiring ground-truth labels.
• Transfer the searched architecture from one task (e.g., denoising) to another (e.g., unpaired image-to-image translation) to evaluate generalization.
• Evaluate performance using standard metrics like PSNR and FID, and conduct user studies to assess perceptual quality.

Experimental results

Research questions

• RQ1 Can neural architecture search discover better upsampling cells than hand-designed ones for deep image prior?
• RQ2 Can learned cross-scale residual connection patterns improve image restoration performance compared to fixed U-Net skip connections?
• RQ3 Does jointly searching over both upsampling cells and connection patterns lead to consistent performance gains across diverse image restoration tasks?
• RQ4 Can a NAS-discovered architecture generalize to learning-based tasks like unpaired image-to-image translation?
• RQ5 How does NAS-DIP compare to state-of-the-art learning-free and learning-based methods in terms of PSNR and perceptual quality?

Key findings

• NAS-DIP achieves a PSNR of 27.84 on the Set14 dataset for 4× super-resolution, a gain of 0.84 over DIP, demonstrating significant improvement in image restoration.
• For 8× super-resolution, NAS-DIP achieves 24.59 PSNR, a 0.44-point improvement over DIP, highlighting the effectiveness of learned upsampling cells.
• In image inpainting, NAS-DIP improves PSNR to 34.72, a 1.24-point gain over DIP, with the largest performance boost observed in this task.
• For image denoising, NAS-DIP reaches 31.42 PSNR, a 0.99-point improvement over DIP, showing consistent gains across tasks.
• When transferred to unpaired image-to-image translation, NAS-DIP reduces FID from 78.62 to 76.22 on the Summer→Winter task, indicating better generative quality.
• User studies show that 79.3% of participants preferred the translated results from NAS-DIP over the standard CycleGAN model, confirming perceptual superiority.