[Paper Review] Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images

Bingna Xu, Yong Guo+3 more|arXiv (Cornell University)|Nov 19, 2022

Seismic Imaging and Inversion Techniques4 citations

TL;DR

This paper proposes Hierarchical Collaborative Downscaling (HCD), a novel method that improves image rescaling by iteratively optimizing both high-resolution (HR) and low-resolution (LR) representations through gradient descent on reconstruction loss. By generating collaborative LR examples via gradient descent, HCD boosts reconstruction performance by up to 1.29 dB PSNR gain over baseline methods without increasing upscaling latency.

ABSTRACT

Deep networks have achieved great success in image rescaling (IR) task that seeks to learn the optimal downscaled representations, i.e., low-resolution (LR) images, to reconstruct the original high-resolution (HR) images. Compared with super-resolution methods that consider a fixed downscaling scheme, e.g., bicubic, IR often achieves significantly better reconstruction performance thanks to the learned downscaled representations. This highlights the importance of a good downscaled representation in image reconstruction tasks. Existing IR methods mainly learn the downscaled representation by jointly optimizing the downscaling and upscaling models. Unlike them, we seek to improve the downscaled representation through a different and more direct way: optimizing the downscaled image itself instead of the down-/upscaling models. Specifically, we propose a collaborative downscaling scheme that directly generates the collaborative LR examples by descending the gradient w.r.t. the reconstruction loss on them to benefit the IR process. Furthermore, since LR images are downscaled from the corresponding HR images, one can also improve the downscaled representation if we have a better representation in the HR domain. Inspired by this, we propose a Hierarchical Collaborative Downscaling (HCD) method that performs gradient descent in both HR and LR domains to improve the downscaled representations. Extensive experiments show that our HCD significantly improves the reconstruction performance both quantitatively and qualitatively. Moreover, we also highlight the flexibility of our HCD since it can generalize well across diverse IR models.

Motivation & Objective

• To improve image rescaling performance by directly optimizing downscaled representations rather than relying solely on joint model training.
• To address the limitation of fixed downscaling kernels in super-resolution by learning better, adaptive downscaled representations.
• To develop a method that enhances reconstruction quality through collaborative learning in both HR and LR domains.
• To ensure the method remains practical by avoiding additional computational cost in the upscaling stage.

Proposed method

• Proposes a collaborative downscaling scheme that performs gradient descent on the LR representation to minimize reconstruction loss, generating improved downscaled examples.
• Introduces a hierarchical optimization framework that alternately optimizes representations in both HR and LR domains to refine the downscaled output.
• Uses a bi-level optimization formulation to model the dependency between HR and LR representations, with iterative refinement in both domains.
• Employs a perturbation budget and inner step size to control the magnitude and stability of updates during collaborative representation learning.
• Generates collaborative examples by descending the gradient w.r.t. reconstruction loss, focusing on high-frequency regions like edges and contours.
• Solves the optimization problem via alternating updates on HR and LR representations, starting from the original image and refining through multiple iterations.

Experimental results

Research questions

• RQ1 Can optimizing the downscaled representation directly, rather than the model parameters, lead to better image reconstruction performance?
• RQ2 How does hierarchical optimization in both HR and LR domains improve the quality of downscaled representations?
• RQ3 What is the impact of iterative refinement on the final reconstruction quality and convergence behavior?
• RQ4 Can the proposed method generalize across diverse image rescaling models without increasing upscaling latency?

Key findings

• HCD achieves a 1.29 dB PSNR improvement over the baseline HCFlow method when generating collaborative LR examples via gradient descent.
• The hierarchical collaborative learning scheme improves performance by 0.22–0.3 dB over using only HR or LR collaborative examples, demonstrating synergy between domains.
• Performance stabilizes after 15 iterations, with negligible gains beyond N=15, indicating convergence within a small number of steps.
• The method increases downscaling latency by only 3.97 seconds for 15 iterations, while introducing no additional latency in the upscaling stage.
• Visualizations show that perturbations are concentrated on high-frequency regions like eyes and butterfly wings, indicating targeted enhancement of fine details.
• The method generalizes well across different IR models, as shown by consistent performance gains on diverse backbones like IRN and HCFlow.