[Paper Review] Priors in Deep Image Restoration and Enhancement: A Survey

Yunfan Lu, Yiqi Lin+5 more|arXiv (Cornell University)|Jun 4, 2022

Advanced Image Processing Techniques4 citations

TL;DR

This survey provides the first comprehensive analysis of priors in deep image restoration and enhancement, categorizing them into structure, statistical, semantic, and deep learning-based priors. It introduces a hierarchical taxonomy, discusses their principles and applications, and identifies future research directions—especially leveraging large-scale foundation models like CLIP and Stable Diffusion—as key enablers for advancing the field.

ABSTRACT

Image restoration and enhancement is a process of improving the image quality by removing degradations, such as noise, blur, and resolution degradation. Deep learning (DL) has recently been applied to image restoration and enhancement. Due to its ill-posed property, plenty of works have been explored priors to facilitate training deep neural networks (DNNs). However, the importance of priors has not been systematically studied and analyzed by far in the research community. Therefore, this paper serves as the first study that provides a comprehensive overview of recent advancements in priors for deep image restoration and enhancement. Our work covers five primary contents: (1) A theoretical analysis of priors for deep image restoration and enhancement; (2) A hierarchical and structural taxonomy of priors commonly used in the DL-based methods; (3) An insightful discussion on each prior regarding its principle, potential, and applications; (4) A summary of crucial problems by highlighting the potential future directions, especially adopting the large-scale foundation models as prior, to spark more research in the community; (5) An open-source repository that provides a taxonomy of all mentioned works and code links.

Motivation & Objective

• To systematically analyze and categorize priors used in deep learning-based image restoration and enhancement.
• To establish a hierarchical and structural taxonomy of priors, including structure, statistical, semantic, and deep learning-based priors.
• To provide an in-depth discussion on the principles, strengths, and applications of each prior type.
• To identify critical challenges and future research directions, particularly the integration of large-scale foundation models as priors.
• To release an open-source repository with a taxonomy of all referenced works and code links for reproducibility.

Proposed method

• Proposes a four-category taxonomy of priors: structure, statistical, semantic, and deep learning-based, with ten subcategories.
• Analyzes the theoretical foundations of priors in the context of ill-posed image restoration problems.
• Reviews prior work in traditional image processing and maps their principles to deep learning frameworks.
• Examines the use of pre-trained GANs and latent space optimization for image generation and restoration.
• Explores the potential of large-scale foundation models (e.g., CLIP, Stable Diffusion, SAM) as semantic priors for zero-shot and few-shot image restoration.
• Introduces a framework for interactive image enhancement using multimodal inputs (e.g., text instructions) via models like Visual ChatGPT.

Experimental results

Research questions

• RQ1 How can priors be systematically classified and structured within deep learning-based image restoration and enhancement?
• RQ2 What are the underlying principles and practical applications of structure, statistical, semantic, and deep learning-based priors?
• RQ3 How can pre-trained GANs and latent code search be leveraged to improve image restoration performance?
• RQ4 What role can large-scale foundation models play in enhancing semantic understanding and fidelity in image restoration?
• RQ5 How can multimodal, instruction-based interfaces improve the accuracy and user personalization of image restoration tasks?

Key findings

• The survey establishes a comprehensive, hierarchical taxonomy of over thirty specific priors across four main categories, enabling systematic comparison and application.
• Structure and statistical priors—originally from classical image processing—remain highly effective when integrated into deep learning frameworks.
• Semantic priors derived from pre-trained models like CLIP and SAM significantly improve zero-shot performance in tasks such as image dehazing and super-resolution.
• Foundation models such as Stable Diffusion and Visual ChatGPT enable interactive, text-guided image restoration, allowing iterative refinement based on user intent.
• GAN inversion techniques using pre-trained generators can generate high-quality images by searching latent codes, offering a data-efficient alternative to supervised training.
• The integration of large-scale foundation models with image restoration tasks is underexploited, representing a major frontier for future research with high potential for performance gains.