[Paper Review] Multi-Scale Denoising in the Feature Space for Low-Light Instance Segmentation

Joanne Lin, Nantheera Anantrasirichai+1 more|arXiv (Cornell University)|Feb 28, 2024

Image Processing Techniques and Applications4 citations

TL;DR

This paper proposes an end-to-end method for low-light instance segmentation by integrating weighted non-local (NL) blocks into the feature extractor of Mask R-CNN, enabling intrinsic feature-level denoising without requiring pre-processing or aligned ground truth. The approach achieves a +10.0 AP improvement over pretrained Mask R-CNN and an additional +1.0 AP with weighted NL blocks, significantly enhancing performance on small and noisy objects in real low-light conditions.

ABSTRACT

Instance segmentation for low-light imagery remains largely unexplored due to the challenges imposed by such conditions, for example shot noise due to low photon count, color distortions and reduced contrast. In this paper, we propose an end-to-end solution to address this challenging task. Our proposed method implements weighted non-local blocks (wNLB) in the feature extractor. This integration enables an inherent denoising process at the feature level. As a result, our method eliminates the need for aligned ground truth images during training, thus supporting training on real-world low-light datasets. We introduce additional learnable weights at each layer in order to enhance the network's adaptability to real-world noise characteristics, which affect different feature scales in different ways. Experimental results on several object detectors show that the proposed method outperforms the pretrained networks with an Average Precision (AP) improvement of at least +7.6, with the introduction of wNLB further enhancing AP by upto +1.3.

Motivation & Objective

• To address the lack of robust instance segmentation methods for real-world low-light sRGB images, which suffer from noise, low contrast, and color distortions.
• To eliminate the need for pre-processing steps like low-light image enhancement (LLIE) by enabling the model to inherently denoise features during training.
• To improve performance on small objects, which are particularly vulnerable to noise in low-light conditions.
• To develop a method that generalizes across different noise characteristics across feature scales using learnable weights in the NL blocks.
• To train effectively on synthetic low-light data without requiring paired normal-light images, enabling use of real-world datasets.

Proposed method

• Integrates weighted non-local (NL) blocks into the backbone of Mask R-CNN to perform feature-level denoising during feature extraction.
• Uses a modified NL operation with learnable weights per feature scale to adaptively suppress noise that varies across spatial and channel dimensions.
• Employs an embedded Gaussian similarity function within the NL block to compute attention-like weights, enhancing feature representation robustness.
• Trains the model end-to-end on a synthetic low-light dataset generated from COCO by simulating low photon counts and reduced illumination.
• Avoids explicit pre-processing by learning to denoise features directly, enabling direct inference on real low-light sRGB images.
• Uses a multi-scale feature fusion strategy with FPN to preserve spatial details, especially for small objects.

(a)

Experimental results

Research questions

• RQ1 Can a deep learning model achieve robust instance segmentation in low-light conditions without relying on pre-processing via LLIE methods?
• RQ2 How effective is feature-level denoising using weighted non-local blocks in improving instance segmentation performance on low-light images?
• RQ3 Does the integration of learnable weights in NL blocks enhance adaptability to scale-specific noise characteristics in feature maps?
• RQ4 How does the proposed method compare to two-stage pipelines that use LLIE followed by instance segmentation in terms of accuracy and robustness?
• RQ5 To what extent does the method improve performance on small objects, which are most affected by noise in low-light conditions?

Key findings

• The proposed method achieves a +10.0 Average Precision (AP) improvement over a standard pretrained Mask R-CNN when trained on synthetic low-light data.
• The addition of weighted NL blocks yields an additional +1.0 AP gain, demonstrating their effectiveness in noise suppression and feature refinement.
• The method outperforms two-stage approaches using state-of-the-art LLIE models (e.g., RetinexFormer, Zero-DCE++) by avoiding noise amplification in enhanced images.
• The embedded Gaussian variant of the NL block performs best among tested variants, achieving 16.6 AP, likely due to its self-attention mechanism.
• The method shows consistent improvements across all AP metrics, including AP_S, AP_M, and AP_L, indicating enhanced robustness for small, medium, and large objects.
• Visual results confirm that the method successfully segments small and distant objects (e.g., small cows in the distance) where baseline models fail, especially under high noise.

(b)