[Paper Review] Algorithms and Theory for Multiple-Source Adaptation

Judy Hoffman, Mehryar Mohri+1 more|arXiv (Cornell University)|May 20, 2018

Domain Adaptation and Few-Shot LearningComputer Science108 citations

TL;DR

The paper proposes normalized distribution-weighted combinations for multiple-source domain adaptation, proves strong theoretical guarantees (including cross-entropy loss) in both deterministic and stochastic settings, provides a DC-programming-based algorithm to compute the mixing weights, and demonstrates robustness and superior empirical performance across several real-world datasets.

ABSTRACT

This work includes a number of novel contributions for the multiple-source adaptation problem. We present new normalized solutions with strong theoretical guarantees for the cross-entropy loss and other similar losses. We also provide new guarantees that hold in the case where the conditional probabilities for the source domains are distinct. Moreover, we give new algorithms for determining the distribution-weighted combination solution for the cross-entropy loss and other losses. We report the results of a series of experiments with real-world datasets. We find that our algorithm outperforms competing approaches by producing a single robust model that performs well on any target mixture distribution. Altogether, our theory, algorithms, and empirical results provide a full solution for the multiple-source adaptation problem with very practical benefits.

Motivation & Objective

• Motivate learning from multiple source domains with no access to the target domain data and no pooling of all sources.
• Extend Mansour et al.'s results to a stochastic setting with a target distribution that is a mixture of source distributions.
• Introduce normalized distribution-weighted mixtures that work with cross-entropy and other losses.
• Provide algorithms to compute mixture weights z via DC-programming with convergence guarantees.
• Demonstrate robustness and practical benefits through extensive experiments on real-world datasets.

Proposed method

• Define distribution-weighted combination h_z^eta for regression and probability models (equations (1) and (2)).
• Show that convex convex combinations are insufficient and that distribution-weighted rules provide favorable guarantees.
• Prove Theorem 1: existence of z and eta such that L(D_T,h_z^eta) is small for target D_T under mild divergence conditions.
• Specialize to cross-entropy loss with normalized predictions, giving a normalized version ar{h}_z^eta and prove its guarantees (Theorem 3).
• Develop a DC-decomposition of the optimization objective and formulate a DC-programming algorithm to find z (Problem (4) and equations (5)-(6)).
• Provide empirical density-estimation-based procedure to implement the approach when source densities are not directly available (Corollary 11).

Experimental results

Research questions

• RQ1 Can a distribution-weighted combination of source predictors achieve small loss on any target mixture without target data?
• RQ2 Do normalization and stochastic extensions preserve strong guarantees for cross-entropy loss?
• RQ3 How can one efficiently compute the optimal mixture weights z in practice?
• RQ4 What is the impact of density estimation errors on the method's theoretical guarantees and empirical performance?

Key findings

• Theorem 1 guarantees existence of a mixture z and margin parameter eta such that the distribution-weighted predictor h_z^eta achieves low loss on the target under mild divergence.
• Corollary 2 shows robustness: when conditional probabilities do not depend on the source, the loss on any mixture is at most epsilon+delta.
• For cross-entropy loss, Theorem 3 provides a normalized distribution-weighted predictor with similar robustness guarantees.
• A DC-programming algorithm converges to a stationary point for finding z, with a practical test to assess optimality.
• Experiments on sentiment analysis, digit recognition, and Office dataset show the distribution-weighted (DW) method outperforms baselines and remains robust across target mixtures, without requiring target data.
• DW competes favorably even against privileged (lambda-comb) or target-aware methods while not needing target labels or retraining.