[Paper Review] Learning unbiased features

Yujia Li, Kevin Swersky+1 more|arXiv (Cornell University)|Dec 17, 2014

Domain Adaptation and Few-Shot Learning11 references17 citations

TL;DR

This paper proposes using Maximum Mean Discrepancy (MMD) as a regularizer to learn unbiased, domain-invariant representations in deep neural networks. By minimizing MMD between source and target domain features, the method improves transfer learning performance across domains, outperforming baselines on sentiment classification and achieving robustness to noise and lighting variations.

ABSTRACT

A key element in transfer learning is representation learning; if representations can be developed that expose the relevant factors underlying the data, then new tasks and domains can be learned readily based on mappings of these salient factors. We propose that an important aim for these representations are to be unbiased. Different forms of representation learning can be derived from alternative definitions of unwanted bias, e.g., bias to particular tasks, domains, or irrelevant underlying data dimensions. One very useful approach to estimating the amount of bias in a representation comes from maximum mean discrepancy (MMD) [5], a measure of distance between probability distributions. We are not the first to suggest that MMD can be a useful criterion in developing representations that apply across multiple domains or tasks [1]. However, in this paper we describe a number of novel applications of this criterion that we have devised, all based on the idea of developing unbiased representations. These formulations include: a standard domain adaptation framework; a method of learning invariant representations; an approach based on noise-insensitive autoencoders; and a novel form of generative model.

Motivation & Objective

• To develop deep learning models that learn representations unbiased to domain shifts, task-specific biases, or irrelevant data variations.
• To address the challenge of limited labeled data in large-scale learning by learning transferable, invariant features across domains.
• To improve generalization in transfer learning by minimizing distributional discrepancies between source and target domains.
• To provide a unified framework using MMD as a regularizer across multiple learning scenarios: supervised, unsupervised, and generative modeling.
• To overcome limitations of existing methods like TCA and denoising autoencoders by using end-to-end MMD-regularized deep networks.

Proposed method

• Uses MMD as a distributional distance measure between feature representations of source and target domains to enforce domain invariance.
• Applies MMD as a penalty term in the loss function of a deep neural network, jointly optimizing feature learning and classification.
• Employs kernel methods to implicitly map features into a reproducing kernel Hilbert space, enabling non-linear distribution matching.
• Integrates MMD into autoencoder training to learn noise-insensitive representations by minimizing MMD between clean and perturbed hidden activations.
• Uses MMD as a direct objective to match the generated data distribution with the real data distribution in generative modeling, avoiding minimax training.
• Leverages backpropagation through the MMD loss to end-to-end train deep models with MMD regularization.

Experimental results

Research questions

• RQ1 Can MMD-based regularization effectively reduce domain shift in transfer learning without requiring labeled target data?
• RQ2 How well can MMD-regularized deep networks learn invariant representations under varying lighting and noise conditions?
• RQ3 Does MMD regularization outperform traditional denoising and contractive autoencoders in learning noise-robust features?
• RQ4 Can MMD be used as a stable, single-objective alternative to minimax training in generative modeling?
• RQ5 To what extent does MMD regularization improve generalization across diverse data distributions in real-world transfer learning scenarios?

Key findings

• The MMD-regularized neural network achieved 79.2% accuracy on the B→D sentiment classification task, outperforming TCA (76.9%) and standard neural nets (78.3%).
• On the K→D task, the MMD model reached 75.0% accuracy with a standard deviation of 1.0%, significantly outperforming the baseline NN (72.7%) and TCA (73.3%).
• The MMD autoencoder achieved a 61.1% SVM accuracy in distinguishing clean from noisy data, indicating superior noise invariance compared to DAE (82.5%) and CAE (77.9%).
• In generative modeling, the MMD-based method successfully learned meaningful features and generated realistic MNIST samples without requiring minimax optimization.
• The MMD-regularized model achieved 85.2% accuracy on the E→K sentiment task, outperforming the best baseline (RBF SVM at 82.0%) and showing robustness across all domain adaptation tasks.
• Surprisingly, denoising autoencoders performed worse than MMD-regularized models in noise invariance, suggesting that denoising does not guarantee robust representations.