[Paper Review] Cross-receptive Focused Inference Network for Lightweight Image Super-Resolution

Wenjie Li, Juncheng Li+4 more|arXiv (Cornell University)|Jul 6, 2022

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes Cross-receptive Focused Inference Network (CFIN), a lightweight hybrid CNN-Transformer architecture for single image super-resolution that enhances feature representation by dynamically fusing cross-scale and contextual information. By integrating a Cross-Scale Information Aggregation Module (CIAM) and a Cross-receptive Field Guided Transformer (CFGT), CFIN achieves state-of-the-art performance with reduced computational cost and model size, outperforming both pure CNN and Transformer baselines on benchmark datasets.

ABSTRACT

Recently, Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks due to the ability of global feature extraction. However, the capabilities of Transformers that need to incorporate contextual information to extract features dynamically are neglected. To address this issue, we propose a lightweight Cross-receptive Focused Inference Network (CFIN) that consists of a cascade of CT Blocks mixed with CNN and Transformer. Specifically, in the CT block, we first propose a CNN-based Cross-Scale Information Aggregation Module (CIAM) to enable the model to better focus on potentially helpful information to improve the efficiency of the Transformer phase. Then, we design a novel Cross-receptive Field Guided Transformer (CFGT) to enable the selection of contextual information required for reconstruction by using a modulated convolutional kernel that understands the current semantic information and exploits the information interaction within different self-attention. Extensive experiments have shown that our proposed CFIN can effectively reconstruct images using contextual information, and it can strike a good balance between computational cost and model performance as an efficient model. Source codes will be available at https://github.com/IVIPLab/CFIN.

Motivation & Objective

• To address the limitation of existing Transformer-based SISR methods in dynamically modeling contextual information for efficient feature extraction.
• To reduce computational complexity and model size while maintaining high image restoration quality in single image super-resolution.
• To bridge the gap between CNNs' local feature extraction and Transformers' global modeling by designing a complementary hybrid architecture.
• To enable efficient deployment on resource-constrained devices through a lightweight, parameter-efficient design.

Proposed method

• Proposes a hybrid network architecture composed of cascaded CT Blocks, each integrating a CNN-based Cross-Scale Information Aggregation Module (CIAM) for early feature refinement.
• Introduces a Cross-receptive Field Guided Transformer (CFGT) that uses modulated convolutional kernels to adaptively select contextually relevant features based on current semantic content.
• Employs a dynamic attention mechanism within the Transformer that leverages inter-self-attention interactions to guide feature selection and enhance long-range dependency modeling.
• Utilizes a cascade of CT Blocks to progressively refine features, balancing local detail preservation and global context integration.
• Designs the CIAM to aggregate features across different scales before the Transformer phase, improving efficiency and focusing on relevant information.
• Optimizes model complexity by balancing parameter count, FLOPs, and inference speed, achieving high performance with minimal computational overhead.

Experimental results

Research questions

• RQ1 Can a hybrid CNN-Transformer architecture effectively combine local and global feature modeling for lightweight SISR?
• RQ2 How can contextual information be dynamically selected and focused during image reconstruction to improve performance?
• RQ3 Can a modulated convolutional kernel in the Transformer attention mechanism enhance feature selection based on semantic context?
• RQ4 Does the proposed CFIN achieve a better trade-off between model efficiency and performance than pure CNN or Transformer baselines?
• RQ5 Can the model maintain high performance under low computational budgets suitable for mobile deployment?

Key findings

• CFIN achieves the highest PSNR of 32.49 dB on Set5 (×4), outperforming both pure-CNN and pure-Transformer models.
• On Urban100 (×2), CFIN achieves 26.39 dB PSNR and 0.7946 SSIM, demonstrating superior performance with minimal computational cost.
• The model achieves a favorable balance between model size (699K parameters), FLOPs (11.45G), and inference time (0.035s), outperforming comparable models in efficiency.
• The ablation study confirms that the combination of CNN and Transformer in CFIN yields better performance than either component alone, validating their complementary nature.
• CFIN achieves state-of-the-art performance on all benchmark datasets (Set5, Set14, Urban100, Manga109) with consistent improvements in PSNR and SSIM.
• The model is deployable on a single NVIDIA RTX 2080Ti GPU and maintains fast inference, making it suitable for mobile and edge applications.