[Paper Review] Domain Generalization via Model-Agnostic Learning of Semantic Features

Qi Dou, Daniel C. Castro+2 more|arXiv (Cornell University)|Oct 29, 2019

Domain Adaptation and Few-Shot LearningComputer Science428 citations

TL;DR

MASF uses episodic model-agnostic training with global class alignment and local metric-based clustering to learn semantic feature spaces that generalize across unseen domains, achieving state-of-the-art results on VLCS and PACS and improvements in medical image segmentation.

ABSTRACT

Generalization capability to unseen domains is crucial for machine learning models when deploying to real-world conditions. We investigate the challenging problem of domain generalization, i.e., training a model on multi-domain source data such that it can directly generalize to target domains with unknown statistics. We adopt a model-agnostic learning paradigm with gradient-based meta-train and meta-test procedures to expose the optimization to domain shift. Further, we introduce two complementary losses which explicitly regularize the semantic structure of the feature space. Globally, we align a derived soft confusion matrix to preserve general knowledge about inter-class relationships. Locally, we promote domain-independent class-specific cohesion and separation of sample features with a metric-learning component. The effectiveness of our method is demonstrated with new state-of-the-art results on two common object recognition benchmarks. Our method also shows consistent improvement on a medical image segmentation task.

Motivation & Objective

• Motivate domain generalization where test domains have unknown statistics and no target data is available during training.
• Learn semantically meaningful feature representations that are robust to domain shifts across multiple source domains.
• Propose global and local regularizers to shape the feature space—global alignment of inter-class relationships and local, domain-independent class clustering.
• Leverage model-agnostic meta-learning with episodic training to induce generalization to unseen domains.

Proposed method

• Adopt episodic training by splitting source domains into meta-train and meta-test to simulate domain shift.
• Introduce a global class alignment loss that matches the soft confusion matrices across meta-train and meta-test domains using symmetric KL divergence.
• Introduce a local sample clustering loss via a metric-embedding network to encourage domain-independent class cohesion and separation using contrastive or triplet losses.
• Update the feature extractor and task net with task loss plus meta loss; update the embedding network with the local loss to enforce clustering.
• Use class-wise mean feature vectors to form soft labels per class and compute a soft confusion matrix via a temperature-controlled softmax, guiding cross-domain semantic alignment.
• Provide two concrete metric-learning losses for local clustering: a contrastive loss (d_phi) and a triplet loss with semi-hard mining for efficient training.

Experimental results

Research questions

• RQ1 How can we train a model on multiple source domains such that it generalizes to unseen domains without access to target domain data during training?
• RQ2 Can explicitly regularizing the semantic structure of the feature space improve domain generalization beyond traditional task-focused losses?
• RQ3 Does combining global inter-class relationship alignment with local sample clustering yield better generalization under domain shift?
• RQ4 Is a model-agnostic episodic learning framework effective on both natural image recognition benchmarks and medical image segmentation tasks?

Key findings

• MASF achieves state-of-the-art average accuracy on VLCS, improving from 72.19 to 74.11 across target domains.
• On PACS, MASF increases average accuracy by 3.51 percentage points over the baseline and shows notable gains on the Sketch target domain.
• Ablation studies confirm that both global class alignment and local clustering contribute to performance, and their combination with episodic training yields the best results.
• Deep residual architectures (ResNet-18/50) also benefit from MASF, demonstrating robustness across different network backbones.
• In medical brain MRI segmentation, MASF improves Dice scores over DeepAll, especially when generalizing to a new clinical site (Set-D), with a Silhouette analysis indicating tighter within-class clustering.