[Paper Review] Multiple Source Adaptation and the Renyi Divergence

Yishay Mansour, Mehryar Mohri+1 more|arXiv (Cornell University)|May 9, 2012

Speech and Audio ProcessingComputer Science17 references73 citations

TL;DR

This paper introduces a theoretical framework for multiple source domain adaptation using Renyi divergence to bound generalization error. It extends prior work by handling arbitrary target distributions, provides loss guarantees under distribution approximation and label shift, and validates performance gains via experiments on synthetic and sentiment data.

ABSTRACT

This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.

Motivation & Objective

• Address the challenge of domain adaptation when multiple source domains are available but the target domain distribution is unknown or not a simple mixture of sources.
• Extend theoretical guarantees beyond previous work that assumed target distributions were mixtures of source distributions.
• Analyze the impact of approximate source distributions and label function shifts on learning performance.
• Provide theoretical bounds on generalization error using Renyi divergence as a measure of distributional discrepancy.
• Validate the effectiveness of distribution-weighted combination strategies in practice through empirical evaluation.

Proposed method

• Formulate the multiple source adaptation problem using Renyi divergence to quantify the discrepancy between source and target distributions.
• Derive generalization bounds that depend on the Renyi divergence between the target distribution and the weighted combination of source distributions.
• Introduce a loss guarantee mechanism that accounts for both known and unknown target distributions.
• Extend the analysis to cases where source distributions are approximated, showing that bounds degrade gracefully with divergence from true distributions.
• Incorporate label shift by modeling differences in labeling functions across domains and adjusting the learning objective accordingly.
• Propose a distribution-weighted combination strategy that optimizes the trade-off between source domain contributions based on their relevance to the target.

Experimental results

Research questions

• RQ1 How can generalization error be bounded when the target distribution is not a mixture of the source distributions?
• RQ2 What is the impact of using approximate source distributions on learning performance in multiple source adaptation?
• RQ3 How do differences in labeling functions across source domains affect the generalization bounds?
• RQ4 Can Renyi divergence serve as an effective measure for selecting optimal source domain weights in domain adaptation?
• RQ5 To what extent do distribution-weighted combinations improve performance on real-world tasks like sentiment analysis?

Key findings

• Theoretical bounds on generalization error are derived using Renyi divergence, showing that smaller divergence leads to tighter bounds.
• The method achieves improved performance on both synthetic data and a sentiment analysis task, demonstrating the benefit of distribution-weighted combinations.
• Even when source distributions are approximated, the loss guarantees remain effective as long as the divergence between true and approximate distributions is bounded.
• The framework handles non-mixture target distributions, significantly broadening the scope beyond prior work.
• Empirical results confirm that the proposed bounds are tight and predictive of actual performance on real datasets.
• The approach remains robust under label shift, with theoretical guarantees that account for differences in labeling functions across domains.