[Paper Review] Multi-Depth Branch Network for Efficient Image Super-Resolution

Huiyuan Tian, Li Zhang+3 more|arXiv (Cornell University)|Sep 29, 2023

Advanced Image Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes a Multi-Depth Branch Network (MDBN) for efficient image super-resolution, using an asymmetric architecture with branches of differing depths to jointly capture high- and low-frequency details. The Multi-Depth Branch Module (MDBM) enables hierarchical feature learning, achieving state-of-the-art (SOTA) performance with inference speeds 1.2× to 18× faster than lightweight transformers, validated through feature visualization and novel Fourier spectral analysis.

ABSTRACT

A longstanding challenge in Super-Resolution (SR) is how to efficiently enhance high-frequency details in Low-Resolution (LR) images while maintaining semantic coherence. This is particularly crucial in practical applications where SR models are often deployed on low-power devices. To address this issue, we propose an innovative asymmetric SR architecture featuring Multi-Depth Branch Module (MDBM). These MDBMs contain branches of different depths, designed to capture high- and low-frequency information simultaneously and efficiently. The hierarchical structure of MDBM allows the deeper branch to gradually accumulate fine-grained local details under the contextual guidance of the shallower branch. We visualize this process using feature maps, and further demonstrate the rationality and effectiveness of this design using proposed novel Fourier spectral analysis methods. Moreover, our model exhibits more significant spectral differentiation between branches than existing branch networks. This suggests that MDBM reduces feature redundancy and offers a more effective method for integrating high- and low-frequency information. Extensive qualitative and quantitative evaluations on various datasets show that our model can generate structurally consistent and visually realistic HR images. It achieves state-of-the-art (SOTA) results at a very fast inference speed. Our code is available at https://github.com/thy960112/MDBN.

Motivation & Objective

• To address the challenge of efficiently enhancing high-frequency details in low-resolution (LR) images while preserving semantic coherence, especially for deployment on low-power devices.
• To overcome the computational inefficiency and high parameter count of existing transformer-based super-resolution models.
• To reduce feature redundancy in branch networks by enabling distinct spectral responses between branches.
• To design a lightweight, efficient CNN-based architecture that outperforms existing methods in both accuracy and inference speed.

Proposed method

• Proposes a Multi-Depth Branch Module (MDBM) with asymmetric branches of different depths, all using 3×3 convolutions for computational efficiency.
• Employs a hierarchical design where the deeper branch captures fine-grained local details under the contextual guidance of the shallower branch.
• Introduces a novel Fourier spectral analysis method to quantify and visualize frequency differentiation between feature maps of different branches.
• Normalizes magnitude spectra of feature maps across branches using min-max scaling to compute spectral difference maps.
• Uses spectral difference maps (D₁ and D₂) to measure divergence in frequency response, demonstrating reduced redundancy.
• Applies feature map visualization and spectral analysis to validate the rationality and effectiveness of the asymmetric branch design.

Experimental results

Research questions

• RQ1 Can an asymmetric, multi-depth branch architecture effectively separate and process high- and low-frequency information in super-resolution?
• RQ2 How does spectral differentiation between branches in MDBM compare to existing symmetric or kernel-size-based branch networks?
• RQ3 To what extent does the hierarchical structure of MDBM improve detail recovery and semantic coherence in reconstructed HR images?
• RQ4 Can the proposed Fourier spectral analysis method provide meaningful insights into frequency-specific feature learning in CNNs?
• RQ5 Does the MDBN achieve SOTA performance with significantly faster inference than state-of-the-art lightweight transformers?

Key findings

• The MDBN achieves state-of-the-art (SOTA) performance on standard benchmarks such as Set5, with PSNR and SSIM scores matching or exceeding those of prior methods.
• The model achieves inference speeds 1.2× to 18× faster than lightweight SR transformers, significantly improving real-time deployment feasibility.
• Spectral analysis reveals that MDBM exhibits more pronounced frequency differentiation between branches than existing branch networks, indicating reduced feature redundancy.
• The distribution of high spectral differences extends beyond the low-frequency center, confirming effective coverage of both high-frequency details and low-frequency contours.
• Feature map visualizations confirm that the deeper branch progressively accumulates fine details under the guidance of the shallower branch’s contextual understanding.
• The proposed Fourier spectral analysis method successfully quantifies and validates the distinct frequency responses of different branches, supporting the design rationale.