[Paper Review] Domain-Irrelevant Representation Learning for Unsupervised Domain Generalization.

Xingxuan Zhang, Linjun Zhou+4 more|arXiv (Cornell University)|Jul 13, 2021

Domain Adaptation and Few-Shot Learning66 references4 citations

TL;DR

This paper proposes Domain-Irrelevant Unsupervised Learning (DIUL), a method for unsupervised domain generalization that learns domain-irrelevant representations from unlabeled data to improve model generalization across distribution shifts. DIUL outperforms ImageNet pretraining even with far less unlabeled data, demonstrating strong robustness and generalization, especially when labeled data are scarce or abundant.

ABSTRACT

Domain generalization (DG) aims to help models trained on a set of source domains generalize better on unseen target domains. The performances of current DG methods largely rely on sufficient labeled data, which however are usually costly or unavailable. While unlabeled data are far more accessible, we seek to explore how unsupervised learning can help deep models generalizes across domains. Specifically, we study a novel generalization problem called unsupervised domain generalization, which aims to learn generalizable models with unlabeled data. Furthermore, we propose a Domain-Irrelevant Unsupervised Learning (DIUL) method to cope with the significant and misleading heterogeneity within unlabeled data and severe distribution shifts between source and target data. Surprisingly we observe that DIUL can not only counterbalance the scarcity of labeled data but also further strengthen the generalization ability of models when the labeled data are sufficient. As a pretraining approach, DIUL shows superior to ImageNet pretraining protocol even when the available data are unlabeled and of a greatly smaller amount compared to ImageNet. Extensive experiments clearly demonstrate the effectiveness of our method compared with state-of-the-art unsupervised learning counterparts.

Motivation & Objective

• To address the challenge of domain generalization in the absence of labeled data, particularly when labeled data are scarce or unavailable.
• To mitigate the impact of domain-specific heterogeneity and severe distribution shifts between source and target domains in unsupervised settings.
• To develop a pretraining method that enhances model generalization beyond standard ImageNet pretraining, even with significantly smaller unlabeled datasets.
• To explore whether unsupervised representation learning can strengthen generalization performance not only in low-data regimes but also when labeled data are plentiful.

Proposed method

• DIUL learns domain-irrelevant representations by minimizing domain-specific features through a contrastive learning objective that encourages invariance across domains.
• The method employs a dual-branch neural network with shared encoder and domain-specific predictors to disentangle domain-specific from generalizable features.
• It uses a domain-invariant contrastive loss that pulls together positive samples (augmentations of the same image) while pushing apart negative samples across domains.
• The framework is trained end-to-end on unlabeled data only, with no reliance on labels during pretraining.
• After pretraining, the model is fine-tuned on downstream tasks with minimal labeled data, leveraging the learned domain-robust features.

Experimental results

Research questions

• RQ1 Can unsupervised representation learning effectively improve domain generalization when labeled data are unavailable?
• RQ2 How does DIUL mitigate the negative impact of domain-specific noise and distribution shifts in unlabeled data?
• RQ3 Does DIUL outperform standard ImageNet pretraining even when using a much smaller amount of unlabeled data?
• RQ4 Can DIUL enhance generalization performance not only in low-data regimes but also when labeled data are abundant?

Key findings

• DIUL achieves superior generalization performance compared to state-of-the-art unsupervised learning methods on multiple domain generalization benchmarks.
• DIUL outperforms ImageNet pretraining even when the available unlabeled data are significantly smaller in scale than ImageNet.
• The method demonstrates improved robustness to distribution shifts, especially in scenarios with limited or no labeled data.
• Surprisingly, DIUL further enhances model generalization even when labeled data are abundant, indicating its effectiveness beyond data-scarce settings.