[Paper Review] A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation

Clayton Scott|arXiv (Cornell University)|Oct 3, 2018

Domain Adaptation and Few-Shot Learning55 references4 citations

TL;DR

This paper introduces a generalized Neyman-Pearson criterion—specifically the Controlled Discovery Rate (CDR)—for optimal domain adaptation in binary classification. It shows that under covariate shift with posterior drift, where source and target distributions differ in feature and label distributions, classifiers trained on source data alone can still optimally minimize the CDR criterion, enabling strong domain adaptation without labeled target data or restrictive similarity assumptions between source and target distributions.

ABSTRACT

In the problem of domain adaptation for binary classification, the learner is presented with labeled examples from a source domain, and must correctly classify unlabeled examples from a target domain, which may differ from the source. Previous work on this problem has assumed that the performance measure of interest is the expected value of some loss function. We introduce a new Neyman-Pearson-like criterion and argue that, for this optimality criterion, stronger domain adaptation results are possible than what has previously been established. In particular, we study a class of domain adaptation problems that generalizes both the covariate shift assumption and a model for feature-dependent label noise, and establish optimal classification on the target domain despite not having access to labelled data from this domain.

Motivation & Objective

• To address the limitations of prior domain adaptation methods that rely on expected loss-based performance measures, which often require strong assumptions about source-target similarity.
• To develop a new optimality criterion—Controlled Discovery Rate (CDR)—that enables stronger domain adaptation guarantees under broader distributional shifts.
• To establish that optimal classification is possible under the combined model of covariate shift and feature-dependent label noise (posterior drift), even without access to labeled target data.
• To demonstrate immunity of the CDR criterion to distributional shifts, allowing source-only training to yield optimal target performance.

Proposed method

• Proposes a generalized Neyman-Pearson (GNP) criterion with parameters θ₁=1 and θ₀=π_Q, leading to the CDR criterion as a special case.
• Defines the discovery rate D_Q(g) = Q_X(g(X)=1) as the constraint, representing the fraction of positive predictions on the target distribution.
• Uses likelihood ratio testing (LRT) as the optimal classifier under the CDR criterion, with thresholding based on the posterior probability η_Q(x).
• Establishes that the optimal classifier under CDR is invariant to posterior drift and covariate shift, enabling immunity to these distributional shifts.
• Introduces two algorithms for CDR optimization in the semi-supervised setting, including a level set method based on kernel logistic regression.
• Proves that the CDR criterion is immune to the combined class of covariate shift with posterior drift, meaning source-only training suffices for optimal target performance.

Experimental results

Research questions

• RQ1 Can a performance criterion other than expected loss yield stronger domain adaptation guarantees under realistic distributional shifts?
• RQ2 Is it possible to achieve optimal domain adaptation under a combined model of covariate shift and posterior drift without labeled target data?
• RQ3 Does the Controlled Discovery Rate (CDR) criterion remain invariant to distributional shifts, enabling source-only training to yield optimal target performance?
• RQ4 How does the CDR criterion compare to traditional 0-1 loss minimization in terms of robustness to feature-dependent label noise?
• RQ5 What are the theoretical and algorithmic foundations for optimizing the CDR criterion in semi-supervised domain adaptation?

Key findings

• The CDR criterion is immune to both covariate shift and posterior drift, meaning optimal classification on the target domain is achievable via source-only training.
• The optimal classifier under CDR is a likelihood ratio test that thresholds the posterior probability η_Q(x), ensuring optimality under the generalized Neyman-Pearson framework.
• The paper establishes the first optimality results for domain adaptation under the combined model of covariate shift and posterior drift, without requiring strong similarity between source and target distributions.
• The CDR criterion enables optimal performance under feature-dependent label noise (FDLN) without the overly restrictive assumptions previously required in the literature.
• Two algorithms are proposed for CDR optimization, including a novel analysis of a level set method based on kernel logistic regression, showing practical feasibility.
• The results generalize prior work on covariate shift and FDLN by providing broader theoretical guarantees under a unified, more realistic distributional model.