[Paper Review] Learning Hybrid Sparsity Prior for Image Restoration: Where Deep Learning Meets Sparse Coding

Wu, Fangfang, Weisheng Dong+2 more|arXiv (Cornell University)|Jul 18, 2018

Advanced Image Processing Techniques5 references4 citations

TL;DR

This paper proposes a hybrid sparsity prior for image restoration by combining external priors learned from deep convolutional neural networks (DCNNs) with internal priors estimated from the input degraded image via structured analysis sparse coding (SASC). The method leverages both large-scale training data and intrinsic image structures to improve restoration performance, achieving state-of-the-art results in denoising and super-resolution tasks.

ABSTRACT

State-of-the-art approaches toward image restoration can be classified into model-based and learning-based. The former - best represented by sparse coding techniques - strive to exploit intrinsic prior knowledge about the unknown high-resolution images; while the latter - popularized by recently developed deep learning techniques - leverage external image prior from some training dataset. It is natural to explore their middle ground and pursue a hybrid image prior capable of achieving the best in both worlds. In this paper, we propose a systematic approach of achieving this goal called Structured Analysis Sparse Coding (SASC). Specifically, a structured sparse prior is learned from extrinsic training data via a deep convolutional neural network (in a similar way to previous learning-based approaches); meantime another structured sparse prior is internally estimated from the input observation image (similar to previous model-based approaches). Two structured sparse priors will then be combined to produce a hybrid prior incorporating the knowledge from both domains. To manage the computational complexity, we have developed a novel framework of implementing hybrid structured sparse coding processes by deep convolutional neural networks. Experimental results show that the proposed hybrid image restoration method performs comparably with and often better than the current state-of-the-art techniques.

Motivation & Objective

• To address the limitations of purely learning-based or model-based image restoration methods by fusing external and internal priors.
• To develop a principled framework that integrates structured sparse coding with deep learning for improved image restoration.
• To reduce computational complexity in hybrid sparse coding by designing a deep network-based implementation.
• To enhance restoration performance on challenging images where training data may not cover all textures or patterns.

Proposed method

• Learns an external structured sparse prior from a large-scale training dataset using a deep convolutional neural network.
• Estimates an internal structured sparse prior directly from the input degraded image using self-similarity and sparse representation.
• Combines the two structured sparse priors into a hybrid prior that integrates knowledge from both external data and internal image structure.
• Designs a novel deep network framework to efficiently implement the hybrid structured sparse coding process.
• Uses a structured analysis sparse coding (SASC) formulation to model image patches with sparsity constraints in a learned transform domain.
• Trains separate networks for different scaling factors in super-resolution tasks using augmented LR/HR patch pairs.

Experimental results

Research questions

• RQ1 Can combining external deep learning priors with internal sparse coding priors improve image restoration performance?
• RQ2 How can a hybrid prior be systematically constructed to leverage both training data and intrinsic image structures?
• RQ3 What is the impact of integrating internal priors on restoring images with rare or absent textures in training data?
• RQ4 Can a deep network efficiently implement hybrid structured sparse coding without sacrificing accuracy?
• RQ5 Does the proposed method outperform state-of-the-art learning-based and model-based approaches in denoising and super-resolution?

Key findings

• The proposed method achieves a PSNR of 27.73 dB on the 'barbara' image in Set10 for image denoising, outperforming EPLL (25.70 dB) and IR-CNN (27.38 dB).
• For super-resolution with scaling factor 3 on Set5, the method achieves a PSNR of 35.56 dB, surpassing SRCNN (35.25 dB), VDSR (35.38 dB), and DRCN (35.50 dB).
• On the Urban100 dataset with scaling factor 4, the method achieves a PSNR of 25.33 dB, outperforming TNRD (24.52 dB), SRCNN (24.52 dB), VDSR (25.14 dB), and DRCN (24.94 dB).
• Visual results demonstrate superior preservation of fine textures and edges, especially in challenging regions like text and fine patterns.
• The hybrid prior framework shows robustness to distribution shifts, performing better than purely learning-based methods on images with rare textures.
• The network-based implementation of SASC enables efficient computation while maintaining high restoration accuracy.