[Paper Review] From Darkness to Detail: Frequency-Aware SSMs for Low-Light Vision

Eashan Adhikarla, Kai Zhang+3 more|arXiv (Cornell University)|Aug 19, 2024

Image Enhancement Techniques4 citations

TL;DR

ExpoMamba proposes a frequency-aware Mamba-based architecture for efficient and effective low-light image enhancement, leveraging a novel Frequency State Space Block (FSSB) to enhance texture details and manage mixed exposure. It achieves 2-3x faster inference (36.6 ms) and up to 20% higher PSNR than prior models, making it suitable for real-time edge deployment.

ABSTRACT

Low-light image enhancement remains a persistent challenge in computer vision, where state-of-the-art models are often hampered by hardware constraints and computational inefficiency, particularly at high resolutions. While foundational architectures like transformers and diffusion models have advanced the field, their computational complexity limits their deployment on edge devices. We introduce ExpoMamba, a novel architecture that integrates a frequency-aware state-space model within a modified U-Net. ExpoMamba is designed to address mixed-exposure challenges by decoupling the modeling of amplitude (intensity) and phase (structure) in the frequency domain. This allows for targeted enhancement, making it highly effective for real-time applications, including downstream tasks like object detection and segmentation. Our experiments on six benchmark datasets show that ExpoMamba is up to 2-3x faster than competing models and achieves a 6.8\% PSNR improvement, establishing a new state-of-the-art in efficient, high-quality low-light enhancement. Source code: https://www.github.com/eashanadhikarla/ExpoMamba.

Motivation & Objective

• Address the challenge of mixed exposure in low-light images, where both underexposed and overexposed regions coexist.
• Overcome the computational inefficiency and high inference latency of transformer- and diffusion-based models in real-time edge applications.
• Develop a lightweight, efficient architecture that maintains high perceptual and structural image quality in low-light enhancement.
• Enable robust multi-resolution inference through a dynamic batch training scheme and adaptive processing of amplitude and phase components.

Proposed method

• Integrate a modified U-Net with 2D-Mamba blocks and frequency domain processing to model long-range dependencies efficiently.
• Propose the Frequency State Space Block (FSSB), which decomposes feature maps into amplitude and phase components for targeted enhancement.
• Apply separate 2D-Mamba blocks to process amplitude (for noise/distortion suppression) and phase (for smoothing and detail recovery).
• Use inverse Fourier transform to reconstruct spatial-domain outputs after frequency-domain processing.
• Implement a dynamic batch training scheme to improve generalization across diverse input resolutions and lighting conditions.
• Apply a composite loss function combining L1, VGG, SSIM, LPIPS, and an overexposed regularization term to preserve structure and avoid artifacts.

Figure 1: [ top: 400x600; bottom: 3840x2160] Scatter plot of model inference time vs. PSNR. Baselines that used ground-truth mean information to produce metrics were reproduced without such information for fairness.

Experimental results

Research questions

• RQ1 Can frequency-aware state space modeling improve low-light image enhancement while maintaining computational efficiency?
• RQ2 How does the separation of amplitude and phase processing in the frequency domain enhance texture recovery and noise suppression?
• RQ3 To what extent does the FSSB block outperform standard convolutional and attention mechanisms in mixed-exposure scenarios?
• RQ4 Can dynamic inference adjustment based on input mean brightness improve performance across diverse lighting conditions?
• RQ5 Does the integration of HDR-aware components (HDR, HDR-CSRNet+) significantly reduce overexposure artifacts in low-light enhancement?

Key findings

• ExpoMamba achieves a PSNR of 25.640 on the LOL-v1 dataset, representing a 15–20% improvement over competing models.
• The model reduces inference time to 36.6 ms, enabling 2–3x faster processing than baseline methods, suitable for real-time applications.
• The FSSB block contributes significantly to performance, with ablation showing a 2.66 dB increase in PSNR when included versus no FSSB.
• Dynamic adjustment (DA) during inference improves PSNR by 0.53 dB and SSIM by 0.015, demonstrating robustness to input brightness variation.
• HDR-CSRNet+ outperforms standard HDR and HDROut layers, achieving the highest PSNR (25.640) and SSIM (0.860) in ablation studies.
• The combination of L1, VGG, SSIM, LPIPS, and overexposure regularization loss leads to superior perceptual quality and artifact suppression.

Figure 2: Overview of the ExpoMamba Architecture. The diagram illustrates the information flow through the ExpoMamba model. The architecture efficiently processes sRGB images by integrating convolutional layers, 2D-Mamba blocks, and deep supervision mechanisms to enhance image reconstruction, partic