[Paper Review] Deep Stacked Hierarchical Multi-patch Network for Image Deblurring

Hongguang Zhang, Yuchao Dai+2 more|arXiv (Cornell University)|Apr 6, 2019

Advanced Image Processing Techniques29 references41 citations

TL;DR

The paper introduces Deep Multi-Patch Hierarchical Network (DMPHN) that uses a fine-to-coarse, non-overlapping patch hierarchy to deblur non-uniform motion blur, achieving real-time 720p deblurring and state-of-the-art PSNR/SSIM on GoPro. It also proposes stacked variants (Stack-DMPHN, Stack-VMPHN) to further boost performance.

ABSTRACT

Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.

Motivation & Objective

• Motivation to address non-uniform motion blur with efficient, scalable deblurring.
• Propose a hierarchical multi-patch CNN inspired by Spatial Pyramid Matching for fine-to-coarse processing.
• Overcome depth-induced performance saturation by introducing stacked multi-patch networks.

Proposed method

• Introduce a hierarchical multi-patch network (DMPHN) with encoder-decoder levels handling non-overlapping patches.
• Process starts at the finest level and progressively contributes residuals to coarser levels via residual-like connections.
• Use an SPM-like patch aggregation strategy without skip/recurrent links to enable small-filter, fast inference.
• Train with an end-to-end loss applied only at the final output level (level 1) to encourage residual-focused learning.
• Explore stacking of DMPHN units (Stack-DMPHN and Stack-VMPHN) to increase performance without prohibitive runtime growth.

Experimental results

Research questions

• RQ1 Can a fine-to-coarse hierarchical multi-patch architecture improve non-uniform motion blur removal more efficiently than multi-scale or scale-recurrent approaches?
• RQ2 Do stacked deployments of DMPHN units yield significant performance gains while maintaining feasible runtimes?
• RQ3 What is the impact of weight sharing across hierarchy levels on performance and model size?

Key findings

• DMPHN achieves state-of-the-art PSNR/SSIM on the GoPro dataset, with the (1-2-4-8) configuration providing best results.
• DMPHN runs in real-time for 720p images (about 30 ms per image at 1280x720).
• Stacked variants (Stack-DMPHN and Stack-VMPHN) deliver over 1 dB PSNR improvements over the base DMPHN on GoPro.
• Stack(4)-DMPHN and Stack(2)-VMPHN further improve results, with Stack variants offering faster-than-comparable performance relative to some baselines.
• The model remains lightweight (e.g., DMPHN encoder/decoder ~3.6 MB) and avoids heavy upsampling/deconvolution between levels.
• Weight sharing reduces parameters with modest PSNR/SSIM trade-offs.