[Paper Review] Modular Vehicle Control for Transferring Semantic Information Between Weather Conditions Using GANs

Patrick Wenzel, Qadeer Khan+2 more|arXiv (Cornell University)|Jul 3, 2018

Autonomous Vehicle Technology and Safety15 references5 citations

TL;DR

This paper proposes a modular vehicle control framework that transfers semantic knowledge across weather conditions using GANs, enabling a single-control module trained on one weather condition to generalize to unseen conditions. By training a perception module via CycleGAN-based unsupervised domain adaptation, the method achieves end-to-end performance comparable to models trained on 15 weather conditions, without collecting new steering labels.

ABSTRACT

Even though end-to-end supervised learning has shown promising results for sensorimotor control of self-driving cars, its performance is greatly affected by the weather conditions under which it was trained, showing poor generalization to unseen conditions. In this paper, we show how knowledge can be transferred using semantic maps to new weather conditions without the need to obtain new ground truth data. To this end, we propose to divide the task of vehicle control into two independent modules: a control module which is only trained on one weather condition for which labeled steering data is available, and a perception module which is used as an interface between new weather conditions and the fixed control module. To generate the semantic data needed to train the perception module, we propose to use a generative adversarial network (GAN)-based model to retrieve the semantic information for the new conditions in an unsupervised manner. We introduce a master-servant architecture, where the master model (semantic labels available) trains the servant model (semantic labels not available). We show that our proposed method trained with ground truth data for a single weather condition is capable of achieving similar results on the task of steering angle prediction as an end-to-end model trained with ground truth data of 15 different weather conditions.

Motivation & Objective

• To address the poor generalization of end-to-end learning in autonomous driving under unseen weather conditions.
• To eliminate the need for collecting new ground truth steering data for each new weather condition.
• To enable transfer of semantic knowledge from a labeled weather condition to unlabeled, unseen conditions using unsupervised learning.
• To decouple perception and control modules so that only the perception module needs to be retrained for new environments.
• To demonstrate that a single control module can achieve robust performance across diverse weather conditions using a GAN-empowered perception interface.

Proposed method

• The vehicle control task is split into two independent modules: a perception module (encoder-decoder) and a control module trained via imitation learning on a single weather condition.
• A master perception module is trained on semantic labels from one weather condition (e.g., default) to extract latent representations from RGB images.
• A servant perception module is trained using CycleGANs to translate images from unseen weather conditions into the domain of the master module, enabling semantic feature extraction without paired data.
• The CycleGANs generate synthetic semantic maps for unseen weather conditions by translating images from those domains to the master domain, using only unpaired images.
• The control module is fixed and only the perception module is replaced when deploying to new weather conditions, ensuring no retraining of the control policy.
• The system uses a master-servant architecture where the master (labeled) trains the servant (unlabeled) via adversarial adaptation, enabling unsupervised semantic transfer.

Experimental results

Research questions

• RQ1 Can a single control policy trained on one weather condition generalize to multiple unseen weather conditions without collecting new labeled data?
• RQ2 How effective is GAN-based unsupervised domain adaptation in preserving semantic structure across diverse weather conditions for autonomous driving?
• RQ3 Can a modular architecture with a fixed control module and replaceable perception module outperform end-to-end learning in zero-shot weather generalization?
• RQ4 What is the impact of CycleGAN-generated semantic maps on steering policy performance in low-visibility or extreme weather conditions?
• RQ5 To what extent can semantic knowledge be transferred from a labeled domain to multiple unlabeled domains using a master-servant perception training framework?

Key findings

• The modular servant model achieved a 96% average success rate across 15 weather conditions, matching the performance of an end-to-end model trained on all 15 conditions.
• The model successfully navigated 100% of turns in sunny and default weather conditions, and maintained over 88% success in 11 out of 15 conditions.
• Performance dropped only on HardRainSunset (condition 13), where low visibility and poor CycleGAN-generated semantics led to 63% success rate.
• The modular approach outperformed end-to-end models trained on only one or a few conditions, especially in conditions far from the training domain.
• The method enabled full generalization to unseen weather conditions without collecting new steering commands or retraining the control module.
• The master-servant GAN framework successfully transferred semantic information across domains in an unsupervised manner, enabling robust control with minimal data dependency.