[Paper Review] ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks

Onur Taşar, S L Happy+2 more|arXiv (Cornell University)|Jul 30, 2019

Generative Adversarial Networks and Image Synthesis74 references33 citations

TL;DR

This paper proposes ColorMapGAN, a novel unsupervised domain adaptation framework for semantic segmentation in remote sensing that uses a lightweight generative adversarial network to transfer spectral distributions from test images to training images via learnable color mapping. By applying a single matrix multiplication and addition, the generator produces semantically consistent yet spectrally adapted fake training images, enabling effective fine-tuning of a pre-trained classifier and achieving state-of-the-art performance with significantly reduced training time compared to existing GAN-based methods.

ABSTRACT

Due to the various reasons such as atmospheric effects and differences in acquisition, it is often the case that there exists a large difference between spectral bands of satellite images collected from different geographic locations. The large shift between spectral distributions of training and test data causes the current state of the art supervised learning approaches to output unsatisfactory maps. We present a novel semantic segmentation framework that is robust to such shift. The key component of the proposed framework is Color Mapping Generative Adversarial Networks (ColorMapGAN), which can generate fake training images that are semantically exactly the same as training images, but whose spectral distribution is similar to the distribution of the test images. We then use the fake images and the ground-truth for the training images to fine-tune the already trained classifier. Contrary to the existing Generative Adversarial Networks (GANs), the generator in ColorMapGAN does not have any convolutional or pooling layers. It learns to transform the colors of the training data to the colors of the test data by performing only one element-wise matrix multiplication and one matrix addition operations. Thanks to the architecturally simple but powerful design of ColorMapGAN, the proposed framework outperforms the existing approaches with a large margin in terms of both accuracy and computational complexity.

Motivation & Objective

• Address the challenge of large domain shifts in spectral distributions between training and test satellite images, which degrade the performance of state-of-the-art CNN-based semantic segmentation models.
• Overcome the limitations of traditional data augmentation and non-learning-based color correction methods that fail under large spectral shifts.
• Develop a computationally efficient yet highly effective method for unsupervised domain adaptation in remote sensing, particularly for semantic segmentation.
• Enable high-quality semantic segmentation maps even when training and test data come from different geographic locations with distinct atmospheric and radiometric conditions.

Proposed method

• Propose ColorMapGAN, a generative adversarial network where the generator performs only one element-wise matrix multiplication and one matrix addition to transform training image colors to match the spectral distribution of test images.
• Design the generator without any convolutional or pooling layers, relying solely on learnable color transformation matrices to preserve semantic consistency while adapting spectral characteristics.
• Train the generator using an adversarial loss to make fake images indistinguishable from real test images in the discriminator, while maintaining semantic fidelity through ground-truth supervision on the original training data.
• Fine-tune the pre-trained semantic segmentation classifier using both real training images and the generated fake images to improve generalization to the target domain.
• Use a discriminator that distinguishes between real test images and fake images generated by the ColorMapGAN, enforcing realistic spectral appearance.
• Implement the framework end-to-end in TensorFlow, optimizing for both performance and training efficiency, with training completed in under 7 minutes on a single GPU.

Experimental results

Research questions

• RQ1 Can a lightweight, non-convolutional generator achieve superior domain adaptation performance in semantic segmentation compared to complex GANs?
• RQ2 Does a learnable color mapping operation via matrix multiplication and addition preserve semantic structure while effectively aligning spectral distributions between domains?
• RQ3 Can the proposed method outperform both non-learning-based methods (e.g., histogram matching, gray world) and deep learning-based image-to-image translation models in accuracy and training efficiency?
• RQ4 To what extent does the proposed framework reduce domain shift-induced performance degradation in semantic segmentation of satellite images?

Key findings

• ColorMapGAN achieves state-of-the-art performance on semantic segmentation tasks across multiple city pairs (Bad Ischl, Villach, Béziers, Roanne), outperforming CycleGAN, UNIT, MUNIT, DRIT, and histogram matching in both accuracy and robustness.
• The method reduces training time to just 0.05 seconds per iteration (approx. 6.5 minutes for 8,000 iterations), significantly faster than CycleGAN (0.11 sec/iter) and other GAN-based methods.
• Unlike histogram matching, which introduces unrealistic color artifacts (e.g., cyan or dark violet rooftops), ColorMapGAN preserves realistic color distributions, as confirmed by visual inspection and histogram analysis.
• ColorMapGAN-generated fake images show more similar class-wise color histograms to real test images than histogram matching, especially for buildings and roads, indicating better spectral alignment.
• The framework maintains high semantic fidelity: buildings with gray or black rooftops in Villach remain unchanged in ColorMapGAN output, avoiding the generation of unnatural colors.
• The approach demonstrates strong generalization, producing accurate predictions even when training and test images exhibit large spectral differences due to time-of-day or atmospheric effects.