[Paper Review] Blind Image Super-Resolution via Contrastive Representation Learning

Jiahui Zhang, Shijian Lu+2 more|arXiv (Cornell University)|Jul 1, 2021

Advanced Image Processing Techniques57 references34 citations

TL;DR

CRL-SR introduces contrastive decoupling encoding and contrastive feature refinement to achieve blind super-resolution under multi-modal and spatially variant degradation without explicit degradation estimation.

ABSTRACT

Image super-resolution (SR) research has witnessed impressive progress thanks to the advance of convolutional neural networks (CNNs) in recent years. However, most existing SR methods are non-blind and assume that degradation has a single fixed and known distribution (e.g., bicubic) which struggle while handling degradation in real-world data that usually follows a multi-modal, spatially variant, and unknown distribution. The recent blind SR studies address this issue via degradation estimation, but they do not generalize well to multi-source degradation and cannot handle spatially variant degradation. We design CRL-SR, a contrastive representation learning network that focuses on blind SR of images with multi-modal and spatially variant distributions. CRL-SR addresses the blind SR challenges from two perspectives. The first is contrastive decoupling encoding which introduces contrastive learning to extract resolution-invariant embedding and discard resolution-variant embedding under the guidance of a bidirectional contrastive loss. The second is contrastive feature refinement which generates lost or corrupted high-frequency details under the guidance of a conditional contrastive loss. Extensive experiments on synthetic datasets and real images show that the proposed CRL-SR can handle multi-modal and spatially variant degradation effectively under blind settings and it also outperforms state-of-the-art SR methods qualitatively and quantitatively.

Motivation & Objective

• Address the challenge of blind SR under real-world, multi-modal, spatially variant degradation.
• Extract resolution-invariant features shared by LR and HR images to mitigate degradation effects.
• Recover lost high-frequency details to produce high-fidelity SR images using learned priors.
• Avoid explicit degradation model estimation by leveraging contrastive representation learning.

Proposed method

• Propose contrastive decoupling encoding (CDE) with two encoders that extract resolution-invariant features from up-sampled LR and HR images under a bidirectional contrastive loss.
• Introduce contrastive feature refinement (CFR) that generates lost high-frequency details guided by a conditional contrastive loss.
• Use a bidirectional InfoNCE-style loss to maximize mutual information between LR and HR invariant features.
• Define a conditional contrastive loss that conditions on resolution-invariant LR features to guide SR feature refinement and image reconstruction.
• Optimize a combined objective including the CFR loss, SR reconstruction loss, and HR reconstruction loss to ensure alignment with HR content.
• Model degradation as I_l = (I_h ⊗ k) ↓_s + n and focus on learning invariant representations rather than estimating k and n.

Experimental results

Research questions

• RQ1 Can a contrastive learning framework decouple resolution-invariant features from degradation in blind SR?
• RQ2 Does conditioning the contrastive refinement on LR invariant features improve recovery of high-frequency details?
• RQ3 How do contrastive decoupling encoding and contrastive feature refinement complement each other in handling multi-modal, spatially variant degradation?
• RQ4 Can the proposed CRL-SR surpass existing blind and non-blind SR methods under complex real-world degradations?

Key findings

• CRL-SR outperforms state-of-the-art SR methods qualitatively and quantitatively on synthetic and real images under blind, multi-modal degradation.
• Ablation studies show that both CDE and CFR contribute to PSNR gains, with the full CRL-SR achieving the best performance.
• The bidirectional contrastive loss effectively extracts resolution-invariant features, reducing degradation impact.
• The conditional contrastive loss enables generation of high-frequency details lost during degradation, improving detail fidelity.
• CRL-SR generalizes to spatially variant degradations and unseen multi-modal degradations better than RCAN, IKC, and DASR in reported experiments.