[论文解读] Non-Local Recurrent Network for Image Restoration

Ding Liu, Bihan Wen+3 more|arXiv (Cornell University)|Jun 7, 2018

Advanced Image Processing Techniques参考文献 51被引用 268

一句话总结

本文提出一个嵌入在循环神经网络中的非局部模块（NLRN），利用图像自相似性的非局部特性进行图像恢复，在参数更少的情况下达到最先进的性能，并在退化条件下表现稳健。

ABSTRACT

Many classic methods have shown non-local self-similarity in natural images to be an effective prior for image restoration. However, it remains unclear and challenging to make use of this intrinsic property via deep networks. In this paper, we propose a non-local recurrent network (NLRN) as the first attempt to incorporate non-local operations into a recurrent neural network (RNN) for image restoration. The main contributions of this work are: (1) Unlike existing methods that measure self-similarity in an isolated manner, the proposed non-local module can be flexibly integrated into existing deep networks for end-to-end training to capture deep feature correlation between each location and its neighborhood. (2) We fully employ the RNN structure for its parameter efficiency and allow deep feature correlation to be propagated along adjacent recurrent states. This new design boosts robustness against inaccurate correlation estimation due to severely degraded images. (3) We show that it is essential to maintain a confined neighborhood for computing deep feature correlation given degraded images. This is in contrast to existing practice that deploys the whole image. Extensive experiments on both image denoising and super-resolution tasks are conducted. Thanks to the recurrent non-local operations and correlation propagation, the proposed NLRN achieves superior results to state-of-the-art methods with much fewer parameters.

研究动机与目标

• 在深度网络中推动将非局部自相似性作为图像恢复的显式先验。
• 提出一个可端到端训练并整合到现有网络中的非局部模块。
• 利用RNN框架提高参数效率并跨状态传播深层特征相关性。
• 通过将相关性计算限制在有限邻域并实现跨状态的相关传播，展示对退化输入的鲁棒性。

提出的方法

• Define a general non-local operation framework that computes Z = diag{delta(X)}^{-1} Phi(X) G(X).
• Introduce a soft block-matching non-local module using learned linear embeddings to compute Phi(X) and G(X) (Equations 4-6).
• Wrap the non-local module with a skip connection for flexible insertion into existing models and enable end-to-end training.
• Embed the non-local module inside a recurrent neural network to share parameters and propagate correlation information across adjacent states (s^t = f_recurrent(s^{t-1}, s^0)).
• Use a 3x3 convolutional backbone with 128 filters, batch normalization, and ReLU activations; train with mean-squared error loss on restoration targets.
• Show that constraining the neighborhood size (q) for correlation improves robustness compared to using the full image.

实验结果

研究问题

• RQ1 Can a non-local operation be effectively integrated into an RNN for image restoration?
• RQ2 Does limiting the correlation neighborhood improve robustness to degraded inputs compared to whole-image non-locality?
• RQ3 Does correlation information propagation across recurrent states enhance restoration performance and training stability?
• RQ4 How does the proposed NLRN perform against state-of-the-art methods in denoising and super-resolution tasks?

主要发现

• The non-local module can be flexibly integrated into deep networks and trained end-to-end for image restoration.
• Using a confined neighborhood for correlation computation improves robustness to degraded inputs.
• Propagating feature correlation between adjacent recurrent states enhances correlation estimation and restoration performance.
• NLRN achieves superior results with significantly fewer parameters compared to several state-of-the-art methods across denoising and super-resolution benchmarks.