[Paper Review] Deep Variational Transfer: Transfer Learning through Semi-supervised Deep Generative Models

Marouan Belhaj, Pavlos Protopapas+1 more|arXiv (Cornell University)|Dec 7, 2018

Domain Adaptation and Few-Shot Learning23 references4 citations

TL;DR

This paper proposes Deep Variational Transfer (DVT), a semi-supervised deep generative model that enables transfer learning across heterogeneous domains by aligning supervised examples of the same class to shared latent Gaussian mixture components, while leveraging unlabeled target data to correct for covariate, prior, and conditional shifts. DVT achieves state-of-the-art performance on MNIST semi-supervised learning and significantly improves F1-scores—doubling them for rare classes—on astronomical datasets like EROS, MACHO, and HiTS.

ABSTRACT

In real-world applications, it is often expensive and time-consuming to obtain labeled examples. In such cases, knowledge transfer from related domains, where labels are abundant, could greatly reduce the need for extensive labeling efforts. In this scenario, transfer learning comes in hand. In this paper, we propose Deep Variational Transfer (DVT), a variational autoencoder that transfers knowledge across domains using a shared latent Gaussian mixture model. Thanks to the combination of a semi-supervised ELBO and parameters sharing across domains, we are able to simultaneously: (i) align all supervised examples of the same class into the same latent Gaussian Mixture component, independently from their domain; (ii) predict the class of unsupervised examples from different domains and use them to better model the occurring shifts. We perform tests on MNIST and USPS digits datasets, showing DVT's ability to perform transfer learning across heterogeneous datasets. Additionally, we present DVT's top classification performances on the MNIST semi-supervised learning challenge. We further validate DVT on a astronomical datasets. DVT achieves states-of-the-art classification performances, transferring knowledge across real stars surveys datasets, EROS, MACHO and HiTS, . In the worst performance, we double the achieved F1-score for rare classes. These experiments show DVT's ability to tackle all major challenges posed by transfer learning: different covariate distributions, different and highly imbalanced class distributions and different feature spaces.

Motivation & Objective

• Address the challenge of transfer learning under general dataset shifts, including covariate, prior, and conditional shifts, in real-world scenarios with limited labeled data.
• Overcome limitations of existing unsupervised transfer learning methods that fail to handle high-dimensional, heterogeneous data with different feature spaces and imbalanced class distributions.
• Enable effective knowledge transfer from labeled source domains to unlabeled target domains by jointly modeling supervised and unsupervised data through a shared latent space.
• Improve classification performance on rare classes in imbalanced datasets by leveraging unlabeled target data to refine latent representations and classifier predictions.

Proposed method

• Propose a variational autoencoder framework with a shared latent Gaussian mixture model (GMM) across source and target domains, where each class is mapped to a unique GMM component.
• Use a semi-supervised evidence lower bound (ELBO) that combines reconstruction loss, KL divergence between posterior and prior, and a classifier loss to jointly train on labeled and unlabeled data.
• Enforce alignment of supervised examples from the same class across domains to the same GMM component via shared parameters in the encoder and classifier networks.
• Leverage domain-specific classifiers $ q_{ heta}(y|oldsymbol{z}) $ trained on latent representations $ oldsymbol{z} $ to predict labels for unlabeled target data and guide their embedding toward appropriate GMM components.
• Apply the Gumbel-Softmax relaxation and reparameterization trick to enable differentiable training of discrete latent variables and gradients through the classifier and latent space.
• Use a straight-through estimator for the Gumbel-softmax to backpropagate gradients through the argmax operation in the classifier, enabling end-to-end optimization of the full model.

Experimental results

Research questions

• RQ1 Can a semi-supervised deep generative model effectively transfer knowledge across domains with different feature spaces, covariate shifts, and class imbalance?
• RQ2 How well can a shared latent GMM structure align examples of the same class across heterogeneous datasets like MNIST and USPS?
• RQ3 To what extent can unlabeled target data improve classification performance, especially for rare classes, in the presence of dataset shift?
• RQ4 Does the proposed method outperform existing state-of-the-art approaches on semi-supervised learning benchmarks and real-world astronomical datasets?
• RQ5 Can the model generalize to complex, high-dimensional data with significant domain shift, such as in multi-survey astronomical object classification?

Key findings

• DVT achieves state-of-the-art performance on the MNIST semi-supervised learning benchmark, demonstrating strong generalization with limited labeled data.
• On astronomical datasets (EROS, MACHO, HiTS), DVT significantly improves classification performance, with the worst-case F1-score for rare classes doubled compared to baseline methods.
• The model effectively handles all three types of dataset shift—covariate, prior, and conditional—by jointly modeling labeled and unlabeled data in a shared latent space.
• By aligning supervised examples of the same class to the same GMM component across domains, DVT reduces domain shift impact and improves zero-shot generalization to unseen target domains.
• The use of unlabeled target data to guide latent space clustering improves model robustness and generalization, especially in low-data regimes and imbalanced settings.
• The Gumbel-Softmax reparameterization enables effective backpropagation through discrete latent variables, allowing end-to-end training of the full model despite non-differentiable operations in the classifier.