[Paper Review] Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction

Zeshuai Deng, Zhuokun Chen+4 more|arXiv (Cornell University)|Oct 29, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes SRTTA, a fast test-time adaptation framework for image super-resolution that addresses domain shift by using a second-order degradation scheme and self-supervised feature reconstruction. It identifies degradation types via a pre-trained classifier, generates pseudo-paired data from second-order degraded images, and adapts the SR model efficiently with minimal computation, achieving state-of-the-art performance on diverse real-world and synthetic degradation types.

ABSTRACT

Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. Conventional SR methods typically gather the paired training data by synthesizing LR images from HR images using a predetermined degradation model, e.g., Bicubic down-sampling. However, the realistic degradation type of test images may mismatch with the training-time degradation type due to the dynamic changes of the real-world scenarios, resulting in inferior-quality SR images. To address this, existing methods attempt to estimate the degradation model and train an image-specific model, which, however, is quite time-consuming and impracticable to handle rapidly changing domain shifts. Moreover, these methods largely concentrate on the estimation of one degradation type (e.g., blur degradation), overlooking other degradation types like noise and JPEG in real-world test-time scenarios, thus limiting their practicality. To tackle these problems, we present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types. Specifically, we design a second-order degradation scheme to construct paired data based on the degradation type of the test image, which is predicted by a pre-trained degradation classifier. Then, we adapt the SR model by implementing feature-level reconstruction learning from the initial test image to its second-order degraded counterparts, which helps the SR model generate plausible HR images. Extensive experiments are conducted on newly synthesized corrupted DIV2K datasets with 8 different degradations and several real-world datasets, demonstrating that our SRTTA framework achieves an impressive improvement over existing methods with satisfying speed. The source code is available at https://github.com/DengZeshuai/SRTTA.

Motivation & Objective

• To address the domain shift problem in super-resolution where test-time degradations differ from training-time degradation models.
• To enable rapid adaptation of pre-trained SR models to unknown or multiple degradation types (e.g., blur, noise, JPEG) without access to source training data.
• To overcome the high inference cost of existing zero-shot and meta-learning-based adaptation methods by minimizing iterative optimization.
• To improve robustness and generalization across diverse real-world degradation types through a self-supervised reconstruction loss.
• To preserve knowledge from the pre-trained model during adaptation to prevent catastrophic forgetting.

Proposed method

• A second-order degradation scheme is proposed to generate pseudo-paired training data: the test image is degraded twice using randomly sampled degradation types to create second-order degraded images.
• A pre-trained degradation classifier is used to identify the primary degradation type in the test image, guiding the selection of second-order degradations.
• A self-supervised second-order reconstruction loss is designed to align features between the test image and its second-order degraded counterparts, enabling effective adaptation without ground-truth HR images.
• Feature-level reconstruction learning is applied to fine-tune the SR model using the pseudo-paired data, allowing the model to learn to reverse multiple degradation types.
• An adaptive parameter preservation strategy is introduced to maintain the pre-trained model’s knowledge during adaptation, reducing forgetting and improving stability.
• The method operates in a single forward pass, making it highly efficient for real-time deployment under rapidly changing degradation conditions.

Experimental results

Research questions

• RQ1 Can a test-time adaptation framework effectively handle unknown or multiple degradation types in super-resolution without access to source training data?
• RQ2 How can pseudo-paired data be efficiently constructed to enable fast and effective adaptation of SR models to diverse degradation types?
• RQ3 Can second-order degradation and reconstruction learning improve the robustness of SR models to complex, real-world degradations compared to first-order or single-degradation approaches?
• RQ4 To what extent does the proposed method reduce adaptation time while maintaining or improving image quality compared to existing zero-shot or meta-learning-based methods?
• RQ5 How well does the method generalize across different degradation types, including blur, noise, and JPEG artifacts, in both synthetic and real-world settings?

Key findings

• SRTTA achieves state-of-the-art performance on the newly constructed DIV2K-C dataset with 8 different degradation types, outperforming existing methods in terms of both PSNR and LPIPS.
• On real-world datasets such as DPED, ADE20K, and OST300, SRTTA consistently generates HR images with fewer artifacts and better preservation of original image details compared to baseline models.
• The method achieves significant performance gains under challenging degradation types like Gaussian blur, noise, and JPEG compression, even when the test image contains multiple degradation types.
• SRTTA reduces adaptation time to near-instant inference speed, enabling real-time deployment in dynamic environments where degradation types change rapidly.
• The ablation study confirms that the second-order degradation scheme and reconstruction loss are critical components, with performance dropping significantly when removed.
• The adaptive parameter preservation strategy effectively mitigates catastrophic forgetting, maintaining high performance across multiple adaptation steps in long-term deployment scenarios.