[Paper Review] Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers

Jingwen He, Chao Dong+1 more|arXiv (Cornell University)|Apr 17, 2019

Advanced Image Processing Techniques21 references4 citations

TL;DR

This paper proposes AdaFM-Net, a unified CNN framework that enables continual, user-controllable modulation of image restoration levels (e.g., denoising, super-resolution, DeJPEG) via adaptive feature modification layers. By training a base model and inserting lightweight AdaFM layers that interpolate filter statistics, the method achieves smooth, artifact-free restoration across arbitrary degradation levels with no retraining, using less than 4% additional parameters and maintaining near-sota performance.

ABSTRACT

In image restoration tasks, like denoising and super resolution, continual modulation of restoration levels is of great importance for real-world applications, but has failed most of existing deep learning based image restoration methods. Learning from discrete and fixed restoration levels, deep models cannot be easily generalized to data of continuous and unseen levels. This topic is rarely touched in literature, due to the difficulty of modulating well-trained models with certain hyper-parameters. We make a step forward by proposing a unified CNN framework that consists of few additional parameters than a single-level model yet could handle arbitrary restoration levels between a start and an end level. The additional module, namely AdaFM layer, performs channel-wise feature modification, and can adapt a model to another restoration level with high accuracy. By simply tweaking an interpolation coefficient, the intermediate model - AdaFM-Net could generate smooth and continuous restoration effects without artifacts. Extensive experiments on three image restoration tasks demonstrate the effectiveness of both model training and modulation testing. Besides, we carefully investigate the properties of AdaFM layers, providing a detailed guidance on the usage of the proposed method.

Motivation & Objective

• Address the gap in deep learning-based image restoration where models are trained on discrete, fixed degradation levels, limiting real-world usability.
• Enable continual, interactive modulation of restoration strength in inference without retraining or model retraining.
• Develop a unified framework that generalizes across continuous degradation levels (e.g., JPEG quality, noise variance) with minimal computational overhead.
• Ensure high output quality across all modulation levels by preserving fidelity at both start and end levels.
• Provide a practical, deployable solution for interactive applications requiring real-time adjustment of restoration effects.

Proposed method

• Introduce the Adaptive Feature Modification (AdaFM) layer, which applies depth-wise convolution to modify feature statistics adaptively during inference.
• Train a base restoration CNN on a start-level degradation (e.g., q80 for DeJPEG), then insert and fine-tune AdaFM layers to adapt to an end-level degradation (e.g., q10).
• During inference, modulate the model by interpolating AdaFM layer filters using a scalar coefficient λ ∈ [0,1], enabling smooth transitions between levels.
• Use a polynomial function λ = Σ w_j * L_c^j to map a continuous degradation level L_c to the interpolation coefficient λ, enabling accurate level prediction.
• Fix all CNN and AdaFM parameters after training, allowing real-time modulation without additional learning or fine-tuning.
• Optimize the AdaFM layers to maintain high performance at both the start and end levels, ensuring robustness across the entire range.

Experimental results

Research questions

• RQ1 Can a single deep learning model be effectively modulated across a continuous range of degradation levels without retraining?
• RQ2 How can feature statistics be modified in a way that enables smooth, artifact-free transitions between restoration levels?
• RQ3 What is the minimal parameter overhead required to achieve high-quality, continuous modulation in image restoration?
• RQ4 How does the performance of the modulated model compare to single-level models at both endpoints of the degradation range?
• RQ5 Can a polynomial mapping between degradation levels and modulation coefficients achieve accurate and stable restoration across unseen levels?

Key findings

• The AdaFM-Net achieves PSNR distances below 0.2 dB from well-trained single-level models across all tested levels, indicating high modulation accuracy.
• Modulation across smaller degradation ranges (e.g., q80→q50) yields significantly lower PSNR distances (below 0.1 dB) than larger ranges (e.g., q80→q10), suggesting better performance with range decomposition.
• The AdaFM layers add less than 4% of the total model parameters while enabling full continuous modulation, making the method highly efficient.
• Qualitative results show no visible artifacts in modulated outputs, even at extreme levels, unlike methods using Conditional IN or AdaBN, which produce noticeable distortions.
• The method successfully enables interactive, real-time adjustment of restoration strength via a single control knob (λ), as demonstrated in DeJPEG, super-resolution, and denoising tasks.
• Curve fitting of the interpolation coefficient λ using polynomial functions achieves accurate mapping between degradation levels and modulation settings, with cubic functions outperforming linear fits on large ranges.