[Paper Review] Attentive Generative Adversarial Network for Raindrop Removal from a Single Image

Rui Qian, Robby T. Tan+3 more|arXiv (Cornell University)|Nov 28, 2017

Image Enhancement Techniques23 references53 citations

TL;DR

The paper proposes an Attentive GAN that learns an attention map to focus on raindrop regions and surrounding areas, removing raindrops from a single image and achieving superior results to prior methods.

ABSTRACT

Raindrops adhered to a glass window or camera lens can severely hamper the visibility of a background scene and degrade an image considerably. In this paper, we address the problem by visually removing raindrops, and thus transforming a raindrop degraded image into a clean one. The problem is intractable, since first the regions occluded by raindrops are not given. Second, the information about the background scene of the occluded regions is completely lost for most part. To resolve the problem, we apply an attentive generative network using adversarial training. Our main idea is to inject visual attention into both the generative and discriminative networks. During the training, our visual attention learns about raindrop regions and their surroundings. Hence, by injecting this information, the generative network will pay more attention to the raindrop regions and the surrounding structures, and the discriminative network will be able to assess the local consistency of the restored regions. This injection of visual attention to both generative and discriminative networks is the main contribution of this paper. Our experiments show the effectiveness of our approach, which outperforms the state of the art methods quantitatively and qualitatively.

Motivation & Objective

• Motivate raindrop removal from a single raindrop-degraded image where raindrop regions and their backgrounds are not provided.
• Introduce an attentive GAN that injects visual attention into both generator and discriminator.
• Develop an attentive-recurrent network to learn raindrop attention and a contextual autoencoder to restore the background.
• Use multi-scale and perceptual losses to improve restoration quality.
• Evaluate the method against state-of-the-art approaches and demonstrate quantitative and qualitative improvements.

Proposed method

• Build a GAN framework with a two-part generator: an attentive-recurrent network that produces an attention map guiding restoration, and a contextual autoencoder that outputs a raindrop-free image.
• Train the attention module using a recurrent block with ResNet-5 blocks and a convolutional LSTM to produce an attention map from 0 to 1.
• Fuse the input image with the final attention map in a contextual autoencoder with skip connections and multi-scale loss functions.
• Apply multi-scale losses on decoder layers to capture contextual information at different scales, plus a perceptual loss using a pretrained network.
• Incorporate an attentive discriminator guided by the attention map to enforce local consistency in restoration, via a map-based loss.
• Leverage a raindrop formation model I = (1-M)⊙B + R to generate training data and an attention-driven loss L_ATT to supervise attention maps.

Experimental results

Research questions

• RQ1 Can an attentive GAN learn to localize raindrop regions and surrounding structures from a single degraded image without explicit raindrop masks during testing?
• RQ2 Does injecting the learned attention map into both the generator and discriminator improve raindrop removal quality compared to non-attentive variants?
• RQ3 Do multi-scale and perceptual losses improve the fidelity of raindrop-free reconstructions?
• RQ4 How does the proposed method compare to existing single-image raindrop removal methods on standard metrics?

Key findings

• The proposed AA+AD architecture achieves higher PSNR and SSIM than Eigen13, Pix2Pix, and ablated variants.
• AA+AD yields PSNR 31.57 and SSIM 0.9023, outperforming all listed baselines.
• Ablation studies show architectures without attention in either generator or discriminator perform worse, validating the need for attentive guidance.
• The method qualitatively removes most raindrops and preserves structural details better than competing methods.
• Quantitative gains are complemented by qualitative examples and improved object recognition scores when applied to vision APIs.