[Paper Review] Omni Aggregation Networks for Lightweight Image Super-Resolution

Hang Wang, Xuanhong Chen+3 more|arXiv (Cornell University)|Apr 20, 2023

Advanced Image Processing Techniques5 citations

TL;DR

This paper proposes Omni-SR, a lightweight vision transformer framework for image super-resolution that introduces Omni Self-Attention (OSA) to simultaneously model spatial and channel-wise feature interactions, and Omni-Scale Aggregation Group (OSAG) to enable hierarchical local, meso-, and global-scale feature learning. The method achieves state-of-the-art performance with only 792K parameters, attaining 26.95 dB on Urban100 ×4 and 31.50 dB on Manga109 ×4.

ABSTRACT

While lightweight ViT framework has made tremendous progress in image super-resolution, its uni-dimensional self-attention modeling, as well as homogeneous aggregation scheme, limit its effective receptive field (ERF) to include more comprehensive interactions from both spatial and channel dimensions. To tackle these drawbacks, this work proposes two enhanced components under a new Omni-SR architecture. First, an Omni Self-Attention (OSA) block is proposed based on dense interaction principle, which can simultaneously model pixel-interaction from both spatial and channel dimensions, mining the potential correlations across omni-axis (i.e., spatial and channel). Coupling with mainstream window partitioning strategies, OSA can achieve superior performance with compelling computational budgets. Second, a multi-scale interaction scheme is proposed to mitigate sub-optimal ERF (i.e., premature saturation) in shallow models, which facilitates local propagation and meso-/global-scale interactions, rendering an omni-scale aggregation building block. Extensive experiments demonstrate that Omni-SR achieves record-high performance on lightweight super-resolution benchmarks (e.g., 26.95 dB@Urban100 $ imes 4$ with only 792K parameters). Our code is available at \url{https://github.com/Francis0625/Omni-SR}.

Motivation & Objective

• To address the limited effective receptive field in lightweight vision transformer-based super-resolution models.
• To overcome the limitations of uni-dimensional self-attention (spatial-only or channel-only) in capturing comprehensive feature interactions.
• To improve feature representation in shallow models by mitigating premature saturation of the effective receptive field.
• To design a multi-scale aggregation scheme that enables progressive, hierarchical feature learning across local, meso-, and global scales.

Proposed method

• Proposes Omni Self-Attention (OSA), a 3D interaction mechanism that simultaneously computes cross-covariance matrices across spatial and channel dimensions to enable omni-axis feature interaction.
• Integrates OSA with window partitioning strategies (e.g., Swin-style) to maintain computational efficiency while enhancing contextual modeling.
• Designs an Omni-Scale Aggregation Group (OSAG) with three cascaded components: local convolution for fine details, meso-scale self-attention for mid-range patterns, and global self-attention for long-range context.
• Uses a hierarchical stacking of distinct operators (conv, self-attention) to avoid the performance saturation seen in homogeneous stacking of identical modules.
• Employs covariance-based channel interaction instead of scalar-based attention (e.g., SE, CBAM), enabling richer feature propagation.
• Maintains model efficiency by controlling FLOPs and parameter count, achieving a compact 792K parameter model.

Experimental results

Research questions

• RQ1 Can simultaneous spatial and channel-wise self-attention improve feature representation in lightweight super-resolution models?
• RQ2 Does a multi-scale aggregation scheme with distinct local, meso-, and global components outperform homogeneous stacking of identical operators?
• RQ3 How does the proposed OSA module compare to conventional attention mechanisms (e.g., SE, CBAM) in terms of performance and optimization stability?
• RQ4 To what extent does the omni-scale design mitigate the effective receptive field saturation problem in shallow models?
• RQ5 Can the proposed framework achieve state-of-the-art performance at a very low parameter count (under 1M)?

Key findings

• Omni-SR achieves 26.95 dB on Urban100 ×4 with only 792K parameters, setting a new SOTA for lightweight models.
• The full Omni-SR model outperforms SwinIR by 0.04 dB on Urban100 ×4, despite having fewer parameters.
• Removing the channel-wise component (Omni-SR_sp) degrades performance by 0.13 dB, demonstrating the importance of channel interaction.
• The OSAG design with all three components (local, meso, global) achieves the best performance, outperforming single or dual-component schemes.
• The full model shows smoother loss landscapes and faster convergence, indicating better optimization properties.
• The OSA module with covariance-based interaction outperforms scalar-based alternatives like SE and CBAM, even when FLOPs are matched.