[Paper Review] Restoring Images in Adverse Weather Conditions via Histogram Transformer

Shangquan Sun, Wenqi Ren+3 more|arXiv (Cornell University)|Jul 14, 2024

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Histoformer, a novel Transformer-based architecture for unified image restoration in adverse weather conditions such as rain, snow, and fog. By introducing histogram self-attention that groups pixels by intensity into dynamic bins and applies self-attention across or within these bins, the model captures long-range spatial dependencies more effectively than conventional spatial or channel-wise attention. The method achieves state-of-the-art performance, with a 0.96 dB PSNR gain over prior methods on benchmark datasets.

ABSTRACT

Transformer-based image restoration methods in adverse weather have achieved significant progress. Most of them use self-attention along the channel dimension or within spatially fixed-range blocks to reduce computational load. However, such a compromise results in limitations in capturing long-range spatial features. Inspired by the observation that the weather-induced degradation factors mainly cause similar occlusion and brightness, in this work, we propose an efficient Histogram Transformer (Histoformer) for restoring images affected by adverse weather. It is powered by a mechanism dubbed histogram self-attention, which sorts and segments spatial features into intensity-based bins. Self-attention is then applied across bins or within each bin to selectively focus on spatial features of dynamic range and process similar degraded pixels of the long range together. To boost histogram self-attention, we present a dynamic-range convolution enabling conventional convolution to conduct operation over similar pixels rather than neighbor pixels. We also observe that the common pixel-wise losses neglect linear association and correlation between output and ground-truth. Thus, we propose to leverage the Pearson correlation coefficient as a loss function to enforce the recovered pixels following the identical order as ground-truth. Extensive experiments demonstrate the efficacy and superiority of our proposed method. We have released the codes in Github.

Motivation & Objective

• To address the limitation of existing Transformer-based methods in capturing long-range spatial features due to fixed-range or channel-only self-attention.
• To unify the restoration of multiple adverse weather conditions—such as rain, snow, and fog—into a single, efficient model.
• To improve feature representation by dynamically grouping pixels based on intensity and applying attention across intensity-based bins.
• To enhance feature learning through a dynamic-range convolution and a dual-scale gated feed-forward module.
• To improve reconstruction fidelity by incorporating a Pearson correlation coefficient-based loss that preserves linear relationships between predicted and ground-truth images.

Proposed method

• Histogram self-attention segments spatial features into intensity-based bins by sorting pixel values in descending order, enabling self-attention across or within these bins to model long-range, weather-related features.
• Two reshaping strategies—bin-wise histogram reshaping (BHR) and frequency-wise histogram reshaping (FHR)—allow the model to balance global integration and fine-grained feature extraction.
• Dynamic-range convolution performs horizontal and vertical pixel sorting before separable convolution, enabling the network to focus on spatially scattered but intensity-similar weather-degraded pixels.
• The Dual-scale Gated Feed-Forward (DGFF) module aggregates features across multiple ranges and scales using pixel shuffling and adaptive gating, enhancing multi-range feature modeling.
• A novel correlation loss based on the Pearson correlation coefficient is introduced to enforce linear ordering consistency between the restored and ground-truth images, improving perceptual quality.
• The Histoformer architecture integrates these components into a unified Transformer backbone, trained end-to-end for joint deraining, dehazing, and desnowing.

(a) Input patches

Experimental results

Research questions

• RQ1 Can histogram-based self-attention improve long-range spatial feature modeling in image restoration under adverse weather conditions compared to fixed-range or channel-only attention?
• RQ2 How does grouping pixels by intensity into dynamic bins enhance the modeling of weather-induced degradation patterns that are spatially scattered but spectrally similar?
• RQ3 To what extent does incorporating a correlation-based loss function improve the structural and linear consistency of restored images compared to standard pixel-wise losses?
• RQ4 How does the dynamic-range convolution improve the learning of non-local, intensity-similar weather artifacts compared to standard convolution?
• RQ5 What is the optimal configuration of bin count and feature dimension for balancing performance and computational cost in the histogram self-attention mechanism?

Key findings

• The proposed Histoformer achieves a 0.96 dB PSNR improvement over the state-of-the-art TKSA method on the SIREN benchmark, demonstrating superior restoration performance.
• The dynamic-range convolution improves PSNR by 0.05 dB compared to vanilla convolution, with the best performance achieved when sorting is applied in horizontal then vertical order.
• Using both BHR and FHR branches in the histogram self-attention module yields the highest PSNR (32.08) and SSIM (0.9389), outperforming single-branch configurations.
• The correlation loss with a weight of 1 achieves the best performance, improving PSNR by 0.31 dB over the baseline without correlation loss.
• The DGFF module achieves a PSNR gain of 0.3 dB over MSFN and 0.76 dB over vanilla FFN, confirming its effectiveness in multi-scale and multi-range feature learning.
• In real-world applications, Histoformer successfully restores snowy scenes, enabling downstream object detection systems to identify previously obscured objects like doors and buildings.

(b) Existing self-attention