[Paper Review] Multiple-Source Adaptation for Regression Problems

Judy Hoffman, Mehryar Mohri+1 more|arXiv (Cornell University)|Nov 14, 2017

Domain Adaptation and Few-Shot Learning24 references3 citations

TL;DR

This paper proposes a distribution-weighted predictor for multiple-source domain adaptation in regression, using DC-programming to optimize squared loss and achieve robust performance across any target mixture of source domains. It demonstrates theoretically and empirically that a single learned model outperforms baselines, including privileged and domain-adapted models, even when the target mixture is unknown.

ABSTRACT

We present a detailed theoretical analysis of the problem of multiple-source adaptation in the general stochastic scenario, extending known results that assume a single target labeling function. Our results cover a more realistic scenario and show the existence of a single robust predictor accurate for \emph{any} target mixture of the source distributions. Moreover, we present an efficient and practical optimization solution to determine the robust predictor in the important case of squared loss, by casting the problem as an instance of DC-programming. We report the results of experiments with both an artificial task and a sentiment analysis task. We find that our algorithm outperforms competing approaches by producing a single robust model that performs well on any target mixture distribution.

Motivation & Objective

• To address the challenge of domain adaptation in regression when only multiple source predictors are available and no target data is accessible.
• To extend theoretical guarantees of multiple-source adaptation from deterministic to stochastic scenarios, covering realistic data distributions.
• To develop an efficient optimization method for learning a single robust predictor that generalizes well across any mixture of source domains.
• To empirically validate the method on artificial and real-world sentiment analysis tasks, showing superiority over existing baselines.

Proposed method

• Formulates the multiple-source adaptation problem as a DC-programming problem under squared loss, enabling efficient optimization.
• Uses distribution-weighted combinations of source predictors, with weights derived from minimizing expected loss over target mixtures.
• Employs empirical estimates of source domain marginals (e.g., bigram language models) to approximate true distributions without labels.
• Applies fixed-point iteration to solve the resulting optimization problem, avoiding the need for target labels.
• Implements a practical algorithm that learns a single predictor valid for any unknown target mixture distribution.
• Uses kernel methods and sample reweighting (KMM) as a baseline, though KMM requires unlabeled target data during training.

Experimental results

Research questions

• RQ1 Can a single robust predictor be learned that performs well across any mixture of multiple source domains in regression?
• RQ2 How can theoretical guarantees of multiple-source adaptation be extended from deterministic to stochastic labeling functions?
• RQ3 What optimization framework enables efficient computation of the distribution-weighted predictor under squared loss?
• RQ4 How does the proposed method compare to baselines when the true target mixture is unknown?

Key findings

• The proposed distribution-weighted (DW) predictor outperforms all baselines, including the privileged λ-comb and KMM, across all tested target mixtures, including single-source and multi-source combinations.
• Even when the target domain is a single source domain, DW achieves lower MSE than the source-specific predictor, demonstrating generalization beyond individual domains.
• In sentiment analysis, DW achieved a mean MSE of 0.142 with standard deviation 0.018 across 10 repetitions, outperforming the uniform combination (0.178) and individual source models.
• The algorithm successfully learns a single robust model without requiring unlabeled target data, unlike KMM, which uses such data for adaptation.
• Experiments confirm that the theoretical robustness of the distribution-weighted predictor holds in practice across diverse mixture distributions.
• The DC-programming formulation enables efficient optimization, making the method practical for real-world regression tasks.