[Paper Review] Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration

Xiaoshuai Zhang, Yiping Lu+2 more|arXiv (Cornell University)|Jan 1, 2018

Image and Signal Denoising Methods35 references22 citations

TL;DR

This paper proposes the Dynamically Unfolding Recurrent Restorer (DURR), a novel image restoration framework that uses a moving endpoint control mechanism to adaptively determine the optimal number of restoration steps for each input image. By modeling image restoration as a dynamic system governed by a recurrent neural network and learning the terminal time via a policy network, DURR achieves state-of-the-art performance in blind image denoising and JPEG deblocking while generalizing robustly to unseen degradation levels beyond the training distribution.

ABSTRACT

In this paper, we propose a new control framework called the moving endpoint control to restore images corrupted by different degradation levels in one model. The proposed control problem contains a restoration dynamics which is modeled by an RNN. The moving endpoint, which is essentially the terminal time of the associated dynamics, is determined by a policy network. We call the proposed model the dynamically unfolding recurrent restorer (DURR). Numerical experiments show that DURR is able to achieve state-of-the-art performances on blind image denoising and JPEG image deblocking. Furthermore, DURR can well generalize to images with higher degradation levels that are not included in the training stage.

Motivation & Objective

• To address the challenge of restoring images with varying degradation levels using a single, unified model.
• To overcome the limitations of fixed-depth networks and manual hyperparameter tuning in existing deep learning-based image restoration methods.
• To improve generalization to unseen degradation levels, such as higher noise levels or unknown quality factors in JPEG compression.
• To develop a unified framework that dynamically adjusts restoration depth per image, enhancing robustness and flexibility.

Proposed method

• The image restoration process is modeled as a continuous-time dynamical system Ẋ = f(X(t), w(t)) using a convolutional recurrent neural network (RNN), which acts as the restoration unit.
• The terminal time τ(x) is not fixed but learned per input image via a policy network, enabling adaptive stopping based on degradation level.
• The framework formulates the problem as a moving endpoint optimal control problem, minimizing a loss function that includes both reconstruction error and regularization.
• The policy network is trained using Deep Q-learning to determine the optimal number of RNN steps (i.e., loop times) for each input image.
• The restoration unit and policy network are trained jointly in an end-to-end manner, with the policy network guiding the dynamic unfolding of the RNN.
• The method is inspired by the connection between residual networks and optimal control, where residual blocks are interpreted as time discretizations of continuous dynamics.

Experimental results

Research questions

• RQ1 Can a single deep learning model effectively restore images with diverse degradation levels without requiring retraining or hyperparameter tuning?
• RQ2 How can the terminal time of a learned image restoration process be made adaptive to input-specific degradation levels?
• RQ3 Does a policy-based dynamic stopping mechanism improve generalization to unseen degradation levels compared to fixed-depth models?
• RQ4 Can the moving endpoint control framework outperform existing state-of-the-art models in blind image denoising and JPEG deblocking?

Key findings

• DURR achieves state-of-the-art performance in blind image denoising, outperforming DnCNN-B and UNLNet5 across all tested noise levels, including unseen levels such as σ = 65 and σ = 75.
• On the BSD68 dataset, DURR achieves a PSNR of 25.26 dB at σ = 65 (unseen during training), while DnCNN-B drops significantly to 23.40 dB, demonstrating strong generalization.
• For JPEG deblocking, DURR outperforms DnCNN-3 in most cases, achieving 34.01 dB at QF 40 and 29.23 dB at QF 10, even though it has significantly fewer parameters.
• In real-world denoising, DURR achieves a PSNR of 22.84 dB on a test image with noise level σ = 65 (unseen in training), surpassing DnCNN’s 21.86 dB and preserving fine textures better.
• The model generalizes well to real-world noise without additional fine-tuning, as shown in qualitative results on images from Lebrun et al. (2015), where DURR restores textures more accurately than baselines.
• The policy network successfully learns to adjust the number of restoration steps per image, enabling the model to handle high-degradation inputs more effectively than fixed-depth alternatives.