[Paper Review] Moment-Based Domain Adaptation: Learning Bounds and Algorithms

Werner Zellinger|arXiv (Cornell University)|Apr 22, 2020

Domain Adaptation and Few-Shot Learning189 references4 citations

TL;DR

This paper proposes a moment-based domain adaptation framework that bounds misclassification risk using finitely many moments and smoothness conditions, enabling robust generalization under weak distributional similarity assumptions. It introduces a novel moment-distance for neural network regularization, outperforming stronger-assumption baselines on benchmark and industrial datasets.

ABSTRACT

This thesis contributes to the mathematical foundation of domain adaptation as emerging field in machine learning. In contrast to classical statistical learning, the framework of domain adaptation takes into account deviations between probability distributions in the training and application setting. Domain adaptation applies for a wider range of applications as future samples often follow a distribution that differs from the ones of the training samples. A decisive point is the generality of the assumptions about the similarity of the distributions. Therefore, in this thesis we study domain adaptation problems under as weak similarity assumptions as can be modelled by finitely many moments.

Motivation & Objective

• Address the challenge of distribution shift in machine learning by relaxing assumptions on distribution similarity.
• Develop a theoretical framework to bound misclassification risk under weak assumptions using finitely many moments and entropy conditions.
• Design practical domain adaptation algorithms based on moment-based metrics that require only unlabeled data from the target domain.
• Demonstrate the effectiveness of the proposed method on large-scale benchmarks and real-world industrial regression problems.

Proposed method

• Propose a moment-distance metric for regularizing neural networks, minimizing the discrepancy between source and target domain moments.
• Introduce a theoretical framework to bound the generalization risk using finite-order moments and smoothness (entropy) conditions on the underlying distributions.
• Design a domain adaptation algorithm that leverages only unlabeled target data to align first-order moments across domains.
• Derive an upper bound on the misclassification risk of the proposed method based on training risk, sample size, moment similarity, and entropy constraints.
• Apply the framework to two industrial problems: time-series modeling in manufacturing and calibration in analytical chemistry.
• Use empirical validation on large-scale benchmarks and real-world datasets to compare performance against standard domain adaptation and regression methods.

Experimental results

Research questions

• RQ1 Can a theoretical risk bound be established for discriminative models under weak assumptions of distribution similarity, specifically finitely many moments?
• RQ2 How can moment-based distance metrics be used to regularize neural networks for improved domain adaptation generalization?
• RQ3 What is the relationship between the proposed moment-distance and existing probability metrics such as Wasserstein or MMD?
• RQ4 Can the proposed method outperform existing domain adaptation approaches that rely on stronger distributional assumptions?
• RQ5 How effective is the method in real-world industrial regression tasks with no labeled target data?

Key findings

• The proposed method achieves lower misclassification risk than baselines when assumptions are relaxed, demonstrating robustness under weak moment similarity.
• Theoretical bounds show that low training risk, large sample size, moment similarity, and entropy conditions jointly ensure low generalization risk.
• The moment-distance metric is shown to be related to other probability metrics, including MMD and Wasserstein, providing theoretical grounding.
• On large-scale benchmarks, the method outperforms existing domain adaptation algorithms that rely on stronger distributional assumptions.
• In industrial manufacturing, the first-moment-based algorithm successfully models time series from unseen distributions, outperforming standard regression models.
• In analytical chemistry, the unlabeled-only calibration method outperforms standard approaches on two real-world datasets, validating its practical utility.