[Paper Review] Domain Generalization via Semi-supervised Meta Learning

Hossein Sharifi-Noghabi, Hossein Asghari+2 more|arXiv (Cornell University)|Sep 26, 2020

Domain Adaptation and Few-Shot Learning38 references6 citations

TL;DR

This paper proposes DGSML, the first domain generalization method that combines semi-supervised learning with meta-learning to leverage both labeled and unlabeled data across multiple source domains with distribution shifts. By using entropy-based pseudo-labeling and novel discrepancy and alignment losses, DGSML achieves state-of-the-art performance on benchmark datasets, especially when labeled data is scarce and unlabeled data is abundant.

ABSTRACT

The goal of domain generalization is to learn from multiple source domains to generalize to unseen target domains under distribution discrepancy. Current state-of-the-art methods in this area are fully supervised, but for many real-world problems it is hardly possible to obtain enough labeled samples. In this paper, we propose the first method of domain generalization to leverage unlabeled samples, combining of meta learning's episodic training and semi-supervised learning, called DGSML. DGSML employs an entropy-based pseudo-labeling approach to assign labels to unlabeled samples and then utilizes a novel discrepancy loss to ensure that class centroids before and after labeling unlabeled samples are close to each other. To learn a domain-invariant representation, it also utilizes a novel alignment loss to ensure that the distance between pairs of class centroids, computed after adding the unlabeled samples, is preserved across different domains. DGSML is trained by a meta learning approach to mimic the distribution shift between the input source domains and unseen target domains. Experimental results on benchmark datasets indicate that DGSML outperforms state-of-the-art domain generalization and semi-supervised learning methods.

Motivation & Objective

• Address the challenge of domain generalization in real-world scenarios where labeled data is scarce but unlabeled data is abundant.
• Overcome the limitation of existing domain generalization methods that rely solely on fully supervised learning and ignore unlabeled samples.
• Develop a method that generalizes to unseen target domains by learning domain-invariant representations using both labeled and unlabeled data.
• Integrate semi-supervised learning techniques—specifically pseudo-labeling and centroid-based consistency—into a meta-learning framework to simulate distribution shift during training.
• Ensure robustness and generalization by minimizing discrepancies between class centroids before and after pseudo-labeling and preserving inter-class centroid distances across domains.

Proposed method

• Employ episodic meta-learning to simulate distribution shifts between source domains and unseen target domains by splitting source domains into meta-train and meta-test sets during each episode.
• Apply entropy-based pseudo-labeling to assign high-confidence predictions as pseudo-labels to unlabeled samples, enabling semi-supervised learning.
• Introduce a discrepancy loss that enforces similarity between class centroids (means) computed before and after pseudo-labeling of unlabeled samples.
• Design an alignment loss that preserves the distance vector between class centroids across different domains after incorporating pseudo-labeled samples.
• Optimize the model using a three-part loss: task-specific cross-entropy loss, semi-supervised discrepancy loss, and domain alignment loss.
• Train the model end-to-end using second-order gradients in a meta-learning setting to improve generalization to unseen domains.

Experimental results

Research questions

• RQ1 Can a domain generalization method effectively leverage both labeled and unlabeled samples from multiple source domains with distribution shifts?
• RQ2 How does combining semi-supervised learning with meta-learning improve generalization to unseen target domains compared to fully supervised domain generalization?
• RQ3 To what extent does pseudo-labeling improve performance when labeled data is limited and unlabeled data is abundant?
• RQ4 Does preserving centroid consistency and inter-domain distance across domains enhance domain-invariant representation learning?
• RQ5 How does the proposed method compare to state-of-the-art methods in terms of accuracy across varying ratios of unlabeled data?

Key findings

• DGSML achieves a test accuracy of 72.98% on the total average benchmark, significantly outperforming the next best method (MASF at 70.09%).
• On the Sketch domain with 50% labeled data, DGSML achieves 95.08% accuracy, compared to 94.12% for the best baseline.
• With ResNet-18 as the feature extractor, DGSML maintains strong performance across all unlabeled data rates, demonstrating robustness.
• When using AlexNet and a low ratio of labeled data (e.g., 50%), DGSML achieves 72.03% accuracy, outperforming all baselines.
• The method shows consistent improvement over state-of-the-art domain generalization and semi-supervised meta-learning methods, especially in low-shot labeled scenarios.
• DGSML trains in under four hours on a single GeForce GTX 1080 GPU, demonstrating computational feasibility despite second-order gradients.