[Paper Review] Restoring Images Captured in Arbitrary Hybrid Adverse Weather Conditions in One Go

Yecong Wan, Mingwen Shao+3 more|arXiv (Cornell University)|May 17, 2023

Image Enhancement Techniques4 citations

TL;DR

This paper proposes RAHC, a unified deep learning framework that restores images degraded by arbitrary hybrid adverse weather conditions (e.g., rain-haze-night) in a single inference pass. By combining a multi-head aggregation block for multi-subspace degradation representation, reconstruction vectors for content guidance, and output-space discrimination for generalization, RAHC achieves state-of-the-art performance on both a new 316K-pair dataset (HAC) and conventional benchmarks, outperforming prior methods by significant margins.

ABSTRACT

Adverse conditions typically suffer from stochastic hybrid weather degradations (e.g., rainy and hazy night), while existing image restoration algorithms envisage that weather degradations occur independently, thus may fail to handle real-world complicated scenarios. Besides, supervised training is not feasible due to the lack of a comprehensive paired dataset to characterize hybrid conditions. To this end, we have advanced the aforementioned limitations with two tactics: framework and data. First, we present a novel unified framework, dubbed RAHC, to Restore Arbitrary Hybrid adverse weather Conditions in one go. Specifically, our RAHC leverages a multi-head aggregation architecture to learn multiple degradation representation subspaces and then constrains the network to flexibly handle multiple hybrid adverse weather in a unified paradigm through a discrimination mechanism in the output space. Furthermore, we devise a reconstruction vectors aided scheme to provide auxiliary visual content cues for reconstruction, thus can comfortably cope with hybrid scenarios with insufficient remaining image constituents. Second, we construct a new dataset, termed HAC, for learning and benchmarking arbitrary Hybrid Adverse Conditions restoration. HAC contains 31 scenarios composed of an arbitrary combination of five common weather, with a total of ~316K adverse-weather/clean pairs. Extensive experiments yield superior results and establish new state-of-the-art results on both HAC and conventional datasets.

Motivation & Objective

• To address the lack of unified models capable of restoring images degraded by arbitrary combinations of multiple adverse weather conditions (e.g., rain, haze, snow, night, raindrops).
• To overcome the limitations of existing methods that treat degradations individually or only for specific pairs, which fails under real-world stochastic hybrid conditions.
• To develop a scalable, data-efficient training pipeline for generating diverse hybrid degradation pairs without expensive manual annotation.
• To establish a comprehensive benchmark dataset (HAC) with ~316K paired images across 31 hybrid weather scenarios to enable rigorous evaluation of generalization and robustness.

Proposed method

• RAHC employs a multi-head blend block (MHBB) that learns multiple degradation representation subspaces using a hybrid of convolution and attention mechanisms, enabling joint modeling of complex, overlapping weather degradations.
• It introduces reconstruction vectors—discrete codebook embeddings pre-trained on natural images—to provide auxiliary visual content cues, improving reconstruction fidelity when image details are severely corrupted.
• A novel output-space discrimination (OSD) mechanism is designed to enforce degradation-invariant restoration capabilities by training a multilabel classifier on the network’s output, promoting generalization across unseen hybrid conditions.
• The HAC dataset is constructed using AdverseGAN, a generative model trained to implicitly learn degradation patterns, enabling automated, low-cost synthesis of diverse hybrid weather pairs.
• The training data are augmented with generated images from AdverseGAN, which are shown to be reliable and effective, reducing annotation cost from ~1 month to ~10 minutes per GPU.
• The framework is compatible with existing SOTA architectures (e.g., Restormer, NAFNet), and OSD can be seamlessly integrated to boost performance in all-in-one restoration settings.

Figure 1: Overview of adverse conditions restoration frameworks. (a) Separate networks designed for specific tasks; (b) generic networks with task-specific weights; (c) unified all-in-one networks with single trained weights; (d) our proposed RAHC framework. In contrast to existing approaches that a

Experimental results

Research questions

• RQ1 Can a single deep learning model effectively restore images degraded by arbitrary combinations of multiple adverse weather conditions, rather than being limited to single or predefined pairs?
• RQ2 How can a model maintain high reconstruction quality when image content is severely degraded and minimal residual structure remains?
• RQ3 To what extent can a generative model like AdverseGAN produce realistic, diverse, and reliable hybrid degradation pairs for training without human-annotated data?
• RQ4 Does output-space discrimination outperform traditional feature-level or knowledge distillation strategies in enabling generalization across unseen hybrid weather scenarios?
• RQ5 Can a unified framework trained on a synthetic, diverse dataset generalize effectively to real-world, complex adverse weather conditions?

Key findings

• RAHC achieves a PSNR of 22.92 on the HAC dataset, significantly outperforming prior SOTA methods such as TKL (21.30) and Restormer+TKL (21.14), demonstrating superior restoration quality.
• The proposed output-space discrimination (OSD) scheme boosts PSNR by up to 0.53 over baseline methods when applied to NAFNet, outperforming the knowledge distillation strategy (TKL) in both PSNR and SSIM gains.
• Models trained on AdverseGAN-generated data achieve comparable performance (PSNR 19.12) to those trained on real data (PSNR 19.16), validating the reliability and efficiency of the synthetic data pipeline.
• The HAC dataset, with 316K paired images across 31 hybrid weather combinations, enables robust evaluation of generalization, revealing that real-world conditions often involve multiple simultaneous degradations.
• Visual results on real-world images show that RAHC produces photorealistic, high-fidelity outputs, confirming its robustness and practicality in real deployment scenarios.
• The integration of reconstruction vectors improves visual quality and structural consistency, especially in low-content scenarios where texture and edge information are largely lost.

Figure 2: Illustration of the proposed RAHC architecture. The restoration network consists of an encoder, a decoder, and a feature mapping subnetwork. The mapping network first maps the encoded feature to the latent clean space and then locates the reconstruction vectors in the pre-established Codeb