[Paper Review] Learning Enriched Features for Real Image Restoration and Enhancement

Syed Waqas Zamir, Aditya Arora+5 more|arXiv (Cornell University)|Mar 15, 2020

Advanced Image Processing Techniques121 references67 citations

TL;DR

MIRNet preserves high-resolution spatial details while fusing multi-scale contextual information through selective kernel feature fusion and dual attention, achieving state-of-the-art results across denoising, super-resolution, and image enhancement on five real-image benchmarks.

ABSTRACT

With the goal of recovering high-quality image content from its degraded version, image restoration enjoys numerous applications, such as in surveillance, computational photography, medical imaging, and remote sensing. Recently, convolutional neural networks (CNNs) have achieved dramatic improvements over conventional approaches for image restoration task. Existing CNN-based methods typically operate either on full-resolution or on progressively low-resolution representations. In the former case, spatially precise but contextually less robust results are achieved, while in the latter case, semantically reliable but spatially less accurate outputs are generated. In this paper, we present a novel architecture with the collective goals of maintaining spatially-precise high-resolution representations through the entire network and receiving strong contextual information from the low-resolution representations. The core of our approach is a multi-scale residual block containing several key elements: (a) parallel multi-resolution convolution streams for extracting multi-scale features, (b) information exchange across the multi-resolution streams, (c) spatial and channel attention mechanisms for capturing contextual information, and (d) attention based multi-scale feature aggregation. In a nutshell, our approach learns an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Extensive experiments on five real image benchmark datasets demonstrate that our method, named as MIRNet, achieves state-of-the-art results for a variety of image processing tasks, including image denoising, super-resolution, and image enhancement. The source code and pre-trained models are available at https://github.com/swz30/MIRNet.

Motivation & Objective

• Preserve high-resolution spatial details throughout the network while incorporating rich contextual information from multiple scales.
• Develop a multi-scale feature extraction and fusion mechanism that jointly leverages fine details and global context.
• Enable dynamic, attention-based aggregation of multi-resolution features with efficient parameter usage.
• Facilitate effective information exchange across parallel resolution streams to improve restoration quality.
• Demonstrate robustness and generalization across diverse real-world datasets and tasks.

Proposed method

• Introduce a multi-scale residual block (MRB) with three parallel streams operating at different resolutions to capture both fine details and contextual information.
• Propose selective kernel feature fusion (SKFF) to dynamically fuse multi-resolution features using self-attention and a fuse-and-select mechanism.
• Incorporate a dual attention unit (DAU) that applies channel and spatial attention to recalibrate features within each stream.
• Use residual resizing modules to perform 2x and 4x downsampling/upsampling while preserving residual learning behavior and shift-equivariance.
• Adopt a recursive residual group (RRG) architecture that stacks MRBs, followed by a final residual prediction and a Charbonnier loss for training.
• Train and evaluate MIRNet on five real-image datasets across denoising, super-resolution, and enhancement tasks, with end-to-end learning (no submodule pre-training).

Experimental results

Research questions

• RQ1 How can high-resolution representations be maintained throughout a restoration network while still leveraging low-resolution context?
• RQ2 Can multi-scale, cross-stream information exchange improve restoration quality without sacrificing spatial precision?
• RQ3 Do selective kernel fusion and dual attention improve cross-scale feature aggregation and overall restoration performance?
• RQ4 How well does the proposed architecture generalize across diverse real-world datasets and tasks (denoising, SR, enhancement)?
• RQ5 What is the impact of architectural components (MRB, SKFF, DAU, residual resizing) on restoration outcomes?

Key findings

• The proposed MIRNet achieves state-of-the-art results on five real image benchmark datasets for denoising, super-resolution, and enhancement.
• SKFF provides effective cross-scale feature fusion with fewer parameters than simple concatenation and improves performance.
• DAU enhances feature representations by applying both channel and spatial attention within streams.
• A recursive residual design with residual resizing enables deep networks while preserving high-resolution details and stable training.
• Ablation studies confirm the importance of skip connections, SKFF, DAU, and the number of streams/columns for performance gains.
• Cross-camera and cross-dataset evaluations demonstrate robust generalization on real-world data.