[Paper Review] Analogical Image Translation for Fog Generation

Rui Gong, Dengxin Dai+3 more|arXiv (Cornell University)|Jun 28, 2020

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Analogical Image Translation (AIT), a zero-shot image-to-image translation framework that transfers fog effects from synthetic clear-weather to real clear-weather images without requiring real foggy images during training. By leveraging supervised learning on synthetic paired data, cycle consistency in the real domain, and adversarial training across domains, the method achieves state-of-the-art performance in generating realistic fog effects, significantly improving downstream semantic foggy scene understanding tasks.

ABSTRACT

Image-to-image translation is to map images from a given \emph{style} to another given \emph{style}. While exceptionally successful, current methods assume the availability of training images in both source and target domains, which does not always hold in practice. Inspired by humans' reasoning capability of analogy, we propose analogical image translation (AIT). Given images of two styles in the source domain: $\mathcal{A}$ and $\mathcal{A}^\prime$, along with images $\mathcal{B}$ of the first style in the target domain, learn a model to translate $\mathcal{B}$ to $\mathcal{B}^\prime$ in the target domain, such that $\mathcal{A}:\mathcal{A}^\prime ::\mathcal{B}:\mathcal{B}^\prime$. AIT is especially useful for translation scenarios in which training data of one style is hard to obtain but training data of the same two styles in another domain is available. For instance, in the case from normal conditions to extreme, rare conditions, obtaining real training images for the latter case is challenging but obtaining synthetic data for both cases is relatively easy. In this work, we are interested in adding adverse weather effects, more specifically fog effects, to images taken in clear weather. To circumvent the challenge of collecting real foggy images, AIT learns with synthetic clear-weather images, synthetic foggy images and real clear-weather images to add fog effects onto real clear-weather images without seeing any real foggy images during training. AIT achieves this zero-shot image translation capability by coupling a supervised training scheme in the synthetic domain, a cycle consistency strategy in the real domain, an adversarial training scheme between the two domains, and a novel network design. Experiments show the effectiveness of our method for zero-short image translation and its benefit for downstream tasks such as semantic foggy scene understanding.

Motivation & Objective

• To address the challenge of limited real-world data for adverse weather conditions, particularly foggy images, in autonomous driving and scene understanding.
• To enable zero-shot image translation where real foggy training images are unavailable, by leveraging analogical reasoning across domains.
• To transfer the translation 'gist' learned from synthetic clear-weather and foggy images to real clear-weather images for realistic fog synthesis.
• To improve the robustness of semantic segmentation models under adverse weather conditions using synthesized fog data.

Proposed method

• The method, named AnalogicalGAN, uses a supervised training scheme on paired synthetic clear-weather and foggy images to learn the translation gist.
• It applies adversarial training between the synthetic and real domains to align the distribution of real clear-weather images with the translated synthetic outputs.
• Cycle consistency loss is enforced in the real domain to ensure that translating the generated foggy image back to clear-weather preserves identity.
• A novel network architecture integrates features from both domains, enabling domain-invariant representation learning for zero-shot translation.
• The model is trained end-to-end with a combination of adversarial, cycle-consistency, and reconstruction losses to ensure realism and fidelity.
• The approach is designed to be generalizable beyond fog generation, applicable to other analogical image translation tasks.

Experimental results

Research questions

• RQ1 Can a model learn to transfer fog effects from synthetic to real images without seeing any real foggy images?
• RQ2 Can analogical reasoning between domains improve zero-shot image translation in low-data regimes?
• RQ3 Does the learned translation gist from synthetic data generalize effectively to real-world image translation?
• RQ4 How does the proposed method compare to physics-based and GAN-based image synthesis in terms of realism and downstream task performance?

Key findings

• AnalogicalGAN outperforms physics-based methods like 'Foggy Cityscapes' and 'Foggy Synscapes' in semantic foggy scene understanding on both Foggy Zurich and Foggy Driving benchmarks.
• On the Foggy Zurich dataset with RefineNet, AnalogicalGAN improves performance by 2.4% over the mixture of 'Foggy Cityscapes' and 'Foggy Synscapes'.
• On the Foggy Driving dataset with BiseNet, the method achieves a 4.7% improvement over the same mixture baseline.
• With RefineNet, AnalogicalGAN reaches 50.3% mIoU on Foggy Driving, comparable to the state-of-the-art 50.7%.
• The method achieves comparable or superior performance to CycleGAN and MUNIT across all test sets and segmentation networks.
• Merging 'AnalogicalGAN Cityscapes' with 'Foggy Synscapes' further boosts performance, demonstrating the method's compatibility and scalability.