[Paper Review] Semi-supervised domain adaptation with CycleGAN guided by a downstream task loss

Annika Mütze, Matthias Rottmann+1 more|arXiv (Cornell University)|Aug 18, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a modular semi-supervised domain adaptation method that guides a CycleGAN generator with a downstream semantic segmentation task loss to reduce domain shift without retraining the segmentation network. By leveraging only 14–70 real-world ground truth images, the method improves mIoU by 4–7 percentage points on Cityscapes and accuracy by 7 points on a classification task, outperforming standard CycleGAN and from-scratch training.

ABSTRACT

Domain adaptation is of huge interest as labeling is an expensive and error-prone task, especially when labels are needed on pixel-level like in semantic segmentation. Therefore, one would like to be able to train neural networks on synthetic domains, where data is abundant and labels are precise. However, these models often perform poorly on out-of-domain images. To mitigate the shift in the input, image-to-image approaches can be used. Nevertheless, standard image-to-image approaches that bridge the domain of deployment with the synthetic training domain do not focus on the downstream task but only on the visual inspection level. We therefore propose a "task aware" version of a GAN in an image-to-image domain adaptation approach. With the help of a small amount of labeled ground truth data, we guide the image-to-image translation to a more suitable input image for a semantic segmentation network trained on synthetic data (synthetic-domain expert). The main contributions of this work are 1) a modular semi-supervised domain adaptation method for semantic segmentation by training a downstream task aware CycleGAN while refraining from adapting the synthetic semantic segmentation expert 2) the demonstration that the method is applicable to complex domain adaptation tasks and 3) a less biased domain gap analysis by using from scratch networks. We evaluate our method on a classification task as well as on semantic segmentation. Our experiments demonstrate that our method outperforms CycleGAN - a standard image-to-image approach - by 7 percent points in accuracy in a classification task using only 70 (10%) ground truth images. For semantic segmentation we can show an improvement of about 4 to 7 percent points in mean Intersection over union on the Cityscapes evaluation dataset with only 14 ground truth images during training.

Motivation & Objective

• To address domain shift in semantic segmentation when transferring models trained on synthetic data to real-world images.
• To reduce reliance on large-scale real-world annotations by using only a small number of labeled real images.
• To develop a modular domain adaptation framework that preserves the pre-trained synthetic-domain segmentation network.
• To enable effective adaptation through task-aware image-to-image translation guided by downstream network performance.
• To provide a less biased domain gap analysis by training all components from scratch, avoiding ImageNet pretraining bias.

Proposed method

• A CycleGAN-based image-to-image translation network is trained to transform real-world images into a synthetic domain distribution.
• The generator is guided by a loss derived from the feature maps of a pre-trained semantic segmentation network, making the translation task-aware.
• The method uses a two-stage training: unsupervised pre-training of the CycleGAN followed by fine-tuning with a small set of real-world ground truth images.
• The downstream task loss is applied during the second stage to align translated images with the semantic segmentation network's expectations.
• The approach maintains the original synthetic-domain segmentation network unchanged, ensuring modularity and avoiding retraining.
• The method is evaluated on both image classification and semantic segmentation tasks using real-world and synthetic data.

Experimental results

Research questions

• RQ1 Can a CycleGAN-based image-to-image translation method be effectively guided by a downstream semantic segmentation task to reduce domain shift?
• RQ2 How much improvement in mIoU and accuracy can be achieved with only a small number of real-world labeled images?
• RQ3 Does training all components from scratch yield a less biased domain gap analysis compared to using ImageNet-pretrained backbones?
• RQ4 Can task-aware translation outperform standard CycleGAN and from-scratch training in complex domain adaptation tasks?
• RQ5 Is there a benefit to using abstract representations (e.g., sketches, simulations) as intermediate domains for improving generalization?

Key findings

• The proposed method improves mean Intersection over Union (mIoU) by 4 to 7 percentage points on the Cityscapes dataset using only 14 real-world ground truth images during training.
• On a classification task, the method achieves a 7 percentage point accuracy gain over standard CycleGAN using only 70 (10%) real-world ground truth images.
• The method significantly reduces domain shift, as evidenced by a 9% mIoU on untranslated real images, which increases to over 40% with the proposed adaptation.
• The use of from-scratch-trained networks for both the generator and segmentation network leads to a less biased domain gap analysis, revealing that ImageNet pretraining may distort gap assessment.
• Longer unsupervised pre-training in stage b) does not substantially improve results, indicating that moderate training is sufficient for effective initialization.
• The method demonstrates strong generalization on complex tasks, including semantic segmentation, even with minimal real-world supervision.