[Paper Review] Principled Hybrids of Generative and Discriminative Domain Adaptation

Han Zhao, Zhenyao Zhu+3 more|arXiv (Cornell University)|May 25, 2017

Domain Adaptation and Few-Shot Learning4 references3 citations

TL;DR

This paper proposes a principled probabilistic framework that unifies generative and discriminative domain adaptation by modeling parameters through flexible priors, enabling joint use of labeled source data and unlabeled source/target data. The framework justifies autoencoder reconstruction loss as an upper bound on marginal likelihood, leading to DAuto—a neural network model that outperforms SOTA methods on text, image, and speech domain adaptation tasks by simultaneously learning discriminative, domain-invariant, and robust representations.

ABSTRACT

We propose a probabilistic framework for domain adaptation that blends both generative and discriminative modeling in a principled way. Under this framework, generative and discriminative models correspond to specific choices of the prior over parameters. This provides us a very general way to interpolate between generative and discriminative extremes through different choices of priors. By maximizing both the marginal and the conditional log-likelihoods, models derived from this framework can use both labeled instances from the source domain as well as unlabeled instances from both source and target domains. Under this framework, we show that the popular reconstruction loss of autoencoder corresponds to an upper bound of the negative marginal log-likelihoods of unlabeled instances, where marginal distributions are given by proper kernel density estimations. This provides a way to interpret the empirical success of autoencoders in domain adaptation and semi-supervised learning. We instantiate our framework using neural networks, and build a concrete model, DAuto. Empirically, we demonstrate the effectiveness of DAuto on text, image and speech datasets, showing that it outperforms related competitors when domain adaptation is possible.

Motivation & Objective

• To address the challenge of domain shift in machine learning, where labeled data in the source domain differs from the target domain.
• To unify generative and discriminative modeling in a single probabilistic framework that allows principled interpolation between the two extremes.
• To provide a theoretical justification for the empirical success of autoencoders in semi-supervised learning and domain adaptation.
• To develop a flexible, end-to-end deep learning model (DAuto) that leverages both labeled and unlabeled data from source and target domains to improve generalization.
• To extend the framework to time-series modeling, demonstrating effectiveness in speech recognition with cross-accent adaptation.

Proposed method

• The framework uses a hierarchical Bayesian model where generative and discriminative models emerge as special cases based on prior distributions over shared and task-specific parameters.
• It jointly maximizes both marginal log-likelihood (for unlabeled data) and conditional log-likelihood (for labeled data), enabling use of unlabeled instances from both domains.
• The reconstruction loss of autoencoders is formally linked to an upper bound on the negative marginal log-likelihood, assuming kernel density estimation for marginal distributions.
• DAuto is instantiated using deep neural networks with shared encoder components, discriminative heads, and reconstruction branches for both source and target domains.
• The model is trained via joint optimization of discriminative, domain-invariant, and reconstruction objectives, using adversarial or MMD-style domain alignment.
• The framework is extended to time-series modeling by adapting the architecture to handle sequential data, validated on cross-accent speech recognition.

Experimental results

Research questions

• RQ1 Can a unified probabilistic framework be designed to smoothly interpolate between generative and discriminative modeling in domain adaptation?
• RQ2 Why do autoencoders improve performance in semi-supervised and domain adaptation settings—can this be explained through a principled probabilistic interpretation?
• RQ3 Can a single deep neural network model effectively learn discriminative, domain-invariant, and robust representations simultaneously using both labeled and unlabeled data?
• RQ4 Does the proposed framework outperform existing SOTA methods in cross-domain transfer learning across diverse data types such as text, images, and speech?
• RQ5 Can the framework be naturally extended to sequential data, such as speech with varying accents, to improve robustness in real-world applications?

Key findings

• DAuto achieves state-of-the-art performance on the Amazon benchmark dataset, outperforming MLP, mSDA, Ladder, and DANN across 16 domain adaptation tasks.
• On the MNIST synthetic experiment, DAuto shows superior performance when domain shift is moderate, confirming its ability to leverage unlabeled target data.
• In speech recognition, DAuto reduces CTC loss by up to 25% on US-CN and CN-US transfers compared to baseline, demonstrating effectiveness in cross-accent adaptation.
• The paired t-test shows that DAuto’s performance gain over DANN is statistically significant (p = 0.072) on average across tasks, with p < 0.05 in 25% of cases.
• When source and target domains are similar (e.g., US vs CN), both DANN and DAuto improve over the baseline, confirming their semi-supervised learning capability.
• Qualitative analysis shows that DAuto maintains robustness even in hard adaptation scenarios where all methods fail, suggesting improved generalization under distribution shift.