[Paper Review] Spatially Adaptive Self-Supervised Learning for Real-World Image Denoising

Junyi Li, Zhilu Zhang+5 more|arXiv (Cornell University)|Mar 27, 2023

Image and Signal Denoising Methods4 citations

TL;DR

This paper proposes a spatially adaptive self-supervised learning framework for real-world image denoising that separately handles flat and textured regions using a blind-neighborhood network (BNN) for flat areas and a locally aware network (LAN) for textured areas. By deriving noise-invariant supervision from non-adjacent pixels in flat regions and content-aware supervision from adjacent pixels in textured regions, the method achieves state-of-the-art performance on SIDD and DND datasets without pixel-shuffle downsampling, avoiding aliasing artifacts and improving both quantitative metrics and perceptual quality.

ABSTRACT

Significant progress has been made in self-supervised image denoising (SSID) in the recent few years. However, most methods focus on dealing with spatially independent noise, and they have little practicality on real-world sRGB images with spatially correlated noise. Although pixel-shuffle downsampling has been suggested for breaking the noise correlation, it breaks the original information of images, which limits the denoising performance. In this paper, we propose a novel perspective to solve this problem, i.e., seeking for spatially adaptive supervision for real-world sRGB image denoising. Specifically, we take into account the respective characteristics of flat and textured regions in noisy images, and construct supervisions for them separately. For flat areas, the supervision can be safely derived from non-adjacent pixels, which are much far from the current pixel for excluding the influence of the noise-correlated ones. And we extend the blind-spot network to a blind-neighborhood network (BNN) for providing supervision on flat areas. For textured regions, the supervision has to be closely related to the content of adjacent pixels. And we present a locally aware network (LAN) to meet the requirement, while LAN itself is selectively supervised with the output of BNN. Combining these two supervisions, a denoising network (e.g., U-Net) can be well-trained. Extensive experiments show that our method performs favorably against state-of-the-art SSID methods on real-world sRGB photographs. The code is available at https://github.com/nagejacob/SpatiallyAdaptiveSSID.

Motivation & Objective

• To address the limitation of existing self-supervised image denoising (SSID) methods in handling spatially correlated noise in real-world sRGB images.
• To overcome the drawbacks of pixel-shuffle downsampling, which breaks image information and causes aliasing artifacts.
• To develop a method that adapts supervision based on local image characteristics—flat vs. textured regions—without requiring clean-noisy pairs.
• To improve denoising performance and efficiency by leveraging spatially adaptive supervision from non-adjacent and adjacent pixels, respectively.
• To train a U-Net-based denoising network using dual supervisions that are content-aware and noise-robust.

Proposed method

• Introduces a blind-neighborhood network (BNN) that extends the blind-spot network (BSN) by expanding the blind region to exclude noise-correlated pixels, enabling reliable supervision in flat regions.
• Uses BNN outputs as a proxy to detect flat regions via high variance, which are then used to supervise the locally aware network (LAN).
• Proposes a locally aware network (LAN) that leverages adjacent pixels to recover texture details, with its supervision selectively guided by BNN outputs in flat areas.
• Employs adaptive coefficients α to spatially fuse BNN and LAN outputs, creating a hybrid supervision signal that respects local image structure.
• Trains a U-Net-based denoising network using the fused, spatially adaptive supervision signal, with BNN and LAN detached during inference.
• Applies a spatially adaptive fusion strategy via learned α, avoiding image-level fusion that degrades performance and increases inference time.

Experimental results

Research questions

• RQ1 Can spatially adaptive supervision improve self-supervised denoising in real-world sRGB images with spatially correlated noise?
• RQ2 How can flat and textured regions be distinguished and processed differently in a self-supervised denoising framework?
• RQ3 Can blind-neighborhood networks effectively suppress noise correlation in flat regions without breaking image structure?
• RQ4 Can locally aware networks leverage adjacent pixels to recover texture details when trained with adaptive supervision from flat regions?
• RQ5 Does spatially adaptive fusion of supervisions outperform image-level fusion in terms of denoising quality and inference efficiency?

Key findings

• The proposed method achieves a PSNR of 37.39 dB on the DND dataset, outperforming state-of-the-art self-supervised methods such as CVF-SID and AP-BSN+R3.
• The ablation study confirms that combining both BNN and LAN supervisions yields the best performance (37.39 dB), with individual components contributing significantly.
• The optimal blind-neighborhood size for BNN is 9×9, which effectively excludes all noise-correlated pixels while preserving signal integrity.
• The locally aware network (LAN) achieves peak performance with a 3×3 local receptive field, balancing detail recovery and blur avoidance.
• The spatially adaptive fusion strategy reduces inference time to 4.8 ms, significantly faster than image-level fusion (22.9 ms), while improving PSNR by 0.55 dB.
• Visual comparisons show that the proposed method produces cleaner results with better preservation of fine textures and less blur compared to baseline methods.