[Paper Review] Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data

Jiaxing Huang, Dayan Guan+2 more|arXiv (Cornell University)|Oct 7, 2021

Domain Adaptation and Few-Shot Learning106 references93 citations

TL;DR

This paper proposes Historical Contrastive Learning (HCL) for unsupervised model adaptation (UMA) that adapts a source-trained model to a target distribution without accessing source data, using historical models to perform instance- and category-level contrastive learning and pseudo-labeling with consistency weighting.

ABSTRACT

Unsupervised domain adaptation aims to align a labeled source domain and an unlabeled target domain, but it requires to access the source data which often raises concerns in data privacy, data portability and data transmission efficiency. We study unsupervised model adaptation (UMA), or called Unsupervised Domain Adaptation without Source Data, an alternative setting that aims to adapt source-trained models towards target distributions without accessing source data. To this end, we design an innovative historical contrastive learning (HCL) technique that exploits historical source hypothesis to make up for the absence of source data in UMA. HCL addresses the UMA challenge from two perspectives. First, it introduces historical contrastive instance discrimination (HCID) that learns from target samples by contrasting their embeddings which are generated by the currently adapted model and the historical models. With the historical models, HCID encourages UMA to learn instance-discriminative target representations while preserving the source hypothesis. Second, it introduces historical contrastive category discrimination (HCCD) that pseudo-labels target samples to learn category-discriminative target representations. Specifically, HCCD re-weights pseudo labels according to their prediction consistency across the current and historical models. Extensive experiments show that HCL outperforms and state-of-the-art methods consistently across a variety of visual tasks and setups.

Motivation & Objective

• Motivate unsupervised model adaptation (UMA) as an alternative to traditional unsupervised domain adaptation when source data cannot be accessed.
• Introduce a memory-based approach that preserves source hypotheses during target-domain adaptation.
• Develop two complementary contrastive mechanisms to learn target representations without source data: instance-discriminative HCID and category-discriminative HCCD.
• Provide theoretical insights and extensive cross-task empirical validation (segmentation, detection, classification) showing HCL's effectiveness over state-of-the-art UMA methods.

Proposed method

• Historical Contrastive Instance Discrimination (HCID): uses embeddings from the current model as queries and embeddings from historical models as keys to perform a memory-based contrastive loss that pulls positives and pushes negatives, with key reliabilities reweighted by historical learning quality.
• Historical Contrastive Category Discrimination (HCCD): pseudo-labels target samples using current and historical predictions, then re-weights self-training loss by historical consistency to learn category-discriminative representations.
• Historical consistency hinge: the pseudo-labels are conditioned on consistency across current and historical models, enabling selective self-training of well-learned samples.
• Theoretical framing links HCID to a maximum likelihood/EM perspective and proves convergence under certain conditions; similarly, HCCD is framed as a classification-ML problem with convergence conditions.

Experimental results

Research questions

• RQ1 Can UMA be effectively achieved without access to source data by leveraging historical source hypotheses?
• RQ2 Do HCID and HCCD provide complementary signals that improve target representation learning under UMA?
• RQ3 How does HCL perform across multiple vision tasks (segmentation, detection, classification) and setups (partial/open-set) compared to state-of-the-art UMA and some UDA baselines?
• RQ4 What is the role of historical consistency in enabling reliable pseudo-labeling and memory-based learning in UMA?

Key findings

• HCL outperforms state-of-the-art UMA methods across semantic segmentation, object detection, and image classification tasks.
• Incorporating HCL (+HCL) consistently improves existing UMA methods, sometimes achieving competitive performance with UDA methods that use source data.
• HCID and HCCD provide orthogonal supervision signals and, when combined, yield the best performance in segmentation tasks.
• HCL demonstrates strong generalization across tasks and learning setups (including partial-set and open-set adaptation).
• Feature visualizations indicate HCL learns both instance-discriminative and category-discriminative representations, preserving source hypotheses while adapting to target data.