[Paper Review] Channel-wise and Spatial Feature Modulation Network for Single Image Super-Resolution

Yanting Hu, Jie Li+2 more|arXiv (Cornell University)|Sep 28, 2018

Advanced Image Processing Techniques40 references20 citations

TL;DR

This paper proposes a Channel-wise and Spatial Feature Modulation (CSFM) network for single image super-resolution, using cascaded Feature-Modulation Memory (FMM) modules with channel-wise and spatial attention residual (CSAR) blocks to dynamically enhance informative features and suppress noise. The gated fusion mechanism preserves long-term context, achieving state-of-the-art performance with fewer parameters than prior methods.

ABSTRACT

The performance of single image super-resolution has achieved significant improvement by utilizing deep convolutional neural networks (CNNs). The features in deep CNN contain different types of information which make different contributions to image reconstruction. However, most CNN-based models lack discriminative ability for different types of information and deal with them equally, which results in the representational capacity of the models being limited. On the other hand, as the depth of neural networks grows, the long-term information coming from preceding layers is easy to be weaken or lost in late layers, which is adverse to super-resolving image. To capture more informative features and maintain long-term information for image super-resolution, we propose a channel-wise and spatial feature modulation (CSFM) network in which a sequence of feature-modulation memory (FMM) modules is cascaded with a densely connected structure to transform low-resolution features to high informative features. In each FMM module, we construct a set of channel-wise and spatial attention residual (CSAR) blocks and stack them in a chain structure to dynamically modulate multi-level features in a global-and-local manner. This feature modulation strategy enables the high contribution information to be enhanced and the redundant information to be suppressed. Meanwhile, for long-term information persistence, a gated fusion (GF) node is attached at the end of the FMM module to adaptively fuse hierarchical features and distill more effective information via the dense skip connections and the gating mechanism. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the superiority of our proposed method over the state-of-the-art methods.

Motivation & Objective

• To address the limited representational capacity of CNNs in super-resolution by enabling discriminative handling of diverse feature types.
• To improve long-term information flow in deep networks by preserving hierarchical features across layers.
• To enhance feature modulation through joint channel-wise and spatial attention in a cascaded residual block design.
• To achieve superior reconstruction quality with reduced model complexity compared to existing methods.

Proposed method

• The CSFM network employs a densely connected architecture with stacked Feature-Modulation Memory (FMM) modules to enhance feature representation.
• Each FMM module contains a chain of Channel-wise and Spatial Attention Residual (CSAR) blocks that apply global-and-local attention to modulate multi-level features.
• The CSAR block integrates channel-wise attention and spatial attention within a residual learning framework to adaptively emphasize important features.
• A Gated Fusion (GF) node is introduced at the end of each FMM module to fuse hierarchical features via learnable gates and dense skip connections.
• The network uses a post-upscaling scheme, processing features directly from low-resolution inputs before final upscaling.
• The model is trained end-to-end using perceptual and reconstruction losses on benchmark datasets.

Experimental results

Research questions

• RQ1 How can deep networks better distinguish and modulate different types of features (e.g., high-frequency vs. low-frequency) in super-resolution?
• RQ2 What mechanisms can effectively preserve long-term contextual information in very deep networks for image restoration?
• RQ3 Can a cascaded attention mechanism improve feature representation without increasing model redundancy?
• RQ4 How does the proposed gated fusion mechanism compare to standard skip connections in preserving hierarchical features?

Key findings

• On the Urban100 dataset, the CSFM network achieves PSNR improvements of 0.19dB, 0.18dB, and 0.14dB over EDSR and RDN at 2×, 3×, and 4× upscaling factors, respectively.
• On the Manga109 dataset, CSFM outperforms EDSR by 0.21dB, 0.32dB, and 0.29dB at 2×, 3×, and 4× scales, demonstrating strong performance on fine-structure images.
• The CSFM model achieves higher PSNR than EDSR and RDN while using 72% and 47% fewer parameters, respectively, showing superior efficiency.
• Visual results show that CSFM reduces artifacts and distortions, producing sharper textures and more recognizable details compared to prior methods.
• The ablation study confirms that both the CSAR blocks and the GF node contribute significantly to performance gains.