[Paper Review] FDA: Fourier Domain Adaptation for Semantic Segmentation

Yanchao Yang, Stefano Soatto|arXiv (Cornell University)|Apr 11, 2020

Domain Adaptation and Few-Shot Learning57 references61 citations

TL;DR

FDA introduces a zero-training domain alignment by swapping low-frequency Fourier amplitudes between source and target images, turning unsupervised domain adaptation into semi-supervised learning and achieving state-of-the-art results on GTA5/SYNTHIA to CityScapes benchmarks.

ABSTRACT

We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other. We illustrate the method in semantic segmentation, where densely annotated images are aplenty in one domain (synthetic data), but difficult to obtain in another (real images). Current state-of-the-art methods are complex, some requiring adversarial optimization to render the backbone of a neural network invariant to the discrete domain selection variable. Our method does not require any training to perform the domain alignment, just a simple Fourier Transform and its inverse. Despite its simplicity, it achieves state-of-the-art performance in the current benchmarks, when integrated into a relatively standard semantic segmentation model. Our results indicate that even simple procedures can discount nuisance variability in the data that more sophisticated methods struggle to learn away.

Motivation & Objective

• Motivate a simple, training-light approach to reduce domain gap in semantic segmentation.
• Leverage low-level Fourier domain statistics to align source and target distributions without adversarial training.
• Demonstrate that FDA can surpass complex domain-adaptation methods on standard benchmarks.
• Integrate FDA into a semi-supervised learning framework to further improve segmentation quality.

Proposed method

• Compute FFT of images and swap the low-frequency amplitude components using a beta-centered mask.
• Reconstruct adapted images with the inverse FFT while keeping the original phase from the source images.
• Train a segmentation network on the adapted source images with cross-entropy loss.
• Regularize with entropy-based weighting on target predictions to encourage confident, well-separated outputs.
• Introduce Multi-band Transfer (MBT) by training multiple models with different beta values and averaging their predictions as pseudo-labels for self-supervision.
• Optional: incorporate entropy minimization (FDA-ENT) and MBT to boost performance.

Experimental results

Research questions

• RQ1 Can simple low-frequency spectral alignment between source and target images reduce domain shift in semantic segmentation without adversarial training?
• RQ2 Does FDA, as a zero-shot alignment method, outperform more complex UDA methods on standard synthetic-to-real benchmarks?
• RQ3 Can combining FDA with entropy-based regularization and MBT yield semi-supervised improvements over the baseline FDA?
• RQ4 How sensitive is FDA to the spectral neighborhood size parameter beta, and can multi-scale/MBT strategies mitigate this sensitivity?

Key findings

• FDA with a single-scale beta achieves strong performance, surpassing several adversarial-domain methods on GTA5→CityScapes and remaining robust across backbones.
• Entropy-regularized FDA (FDA-ENT) yields competitive results similar to more complex alignment approaches.
• Multi-band Transfer (MBT), which averages predictions from models trained with different beta values, provides consistent improvements over single beta models.
• Self-supervised rounds using pseudo-labels from MBT further improve performance, with regularization preventing confirmation bias.
• Across benchmarks and backbones, FDA-based methods achieve top or competitive mIoU scores compared to state-of-the-art methods reliant on image translation or adversarial training.