[Paper Review] Test-time Fourier Style Calibration for Domain Generalization

Xingchen Zhao, Chang Liu+3 more|arXiv (Cornell University)|May 13, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Test-time Fourier Style Calibration (TF-Cal), a method that calibrates target domain style features using source domain prototypes via Fourier decomposition at inference time, while introducing Augment Amplitude Features (AAF) to enhance style diversity. The approach achieves state-of-the-art performance on image classification and medical image segmentation benchmarks by reducing domain shift through on-the-fly style alignment without requiring target domain data during training.

ABSTRACT

The topic of generalizing machine learning models learned on a collection of source domains to unknown target domains is challenging. While many domain generalization (DG) methods have achieved promising results, they primarily rely on the source domains at train-time without manipulating the target domains at test-time. Thus, it is still possible that those methods can overfit to source domains and perform poorly on target domains. Driven by the observation that domains are strongly related to styles, we argue that reducing the gap between source and target styles can boost models' generalizability. To solve the dilemma of having no access to the target domain during training, we introduce Test-time Fourier Style Calibration (TF-Cal) for calibrating the target domain style on the fly during testing. To access styles, we utilize Fourier transformation to decompose features into amplitude (style) features and phase (semantic) features. Furthermore, we present an effective technique to Augment Amplitude Features (AAF) to complement TF-Cal. Extensive experiments on several popular DG benchmarks and a segmentation dataset for medical images demonstrate that our method outperforms state-of-the-art methods.

Motivation & Objective

• To address domain generalization in deep learning where models fail on unseen target domains due to domain shift.
• To overcome the limitation of existing domain generalization methods that rely solely on source domain data during training and cannot adapt to unknown target domains.
• To reduce the domain gap by aligning target domain styles with source domain styles during inference, leveraging Fourier decomposition of features.
• To improve model generalizability through a simple yet effective test-time calibration and feature augmentation mechanism that does not require domain labels.

Proposed method

• The method decomposes early convolutional features into amplitude (style) and phase (semantic) components using Fourier transformation.
• Test-time Fourier Style Calibration (TF-Cal) convexly calibrates the target domain's amplitude features using a prototype derived from source domain features.
• The source domain prototype is estimated during training and used at inference to align target domain styles with source domain styles.
• Augment Amplitude Features (AAF) diversifies higher-level style representations by augmenting amplitude features in the feature space, improving robustness.
• TF-Cal and AAF are combined into TAF-Cal, which is applied at test time without requiring domain labels or target domain data during training.
• The approach is architecture-agnostic and can be integrated into any CNN backbone.

Experimental results

Research questions

• RQ1 Can on-the-fly style calibration at test time improve generalization to unseen target domains in domain generalization?
• RQ2 How does Fourier decomposition of features into amplitude (style) and phase (semantic) components enable effective domain gap reduction?
• RQ3 Can augmenting amplitude features in the feature space improve model generalizability beyond image-space augmentation?
• RQ4 What is the optimal location and hyperparameter configuration for applying TF-Cal within a CNN architecture?
• RQ5 How do TF-Cal and AAF complement each other in boosting performance across diverse benchmarks?

Key findings

• TAF-Cal achieves state-of-the-art performance on the PACS benchmark, outperforming previous SOTA methods with a top-1 accuracy of 86.73% on ResNet-50.
• TF-Cal alone achieves 85.70% top-1 accuracy on PACS with ResNet-18, surpassing most existing methods.
• The ablation study confirms that TF-Cal must be applied at test time to achieve performance gains; applying it only at training time yields no significant improvement.
• AAF alone achieves 85.69% accuracy on PACS, outperforming FACT (83.44%) and demonstrating the effectiveness of feature-space amplitude augmentation.
• The optimal hyperparameter setting for TF-Cal is η = τ = 0.5, as larger values degrade performance by over-calibrating target styles.
• t-SNE visualization confirms that TF-Cal produces distinct, well-separated clusters for target domains, while AAF alone leads to blurred boundaries, showing that TF-Cal enhances both alignment and discriminability.