[Paper Review] A Recent Survey of Heterogeneous Transfer Learning

Runxue Bao, Yiming Sun+5 more|arXiv (Cornell University)|Oct 12, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper presents a comprehensive, up-to-date survey of heterogeneous transfer learning (HTL), covering recent advances in methods, challenges, and applications since 2017. It systematically reviews data-based and model-based HTL techniques, emphasizes the role of pre-trained models and multimodal learning, and identifies key research gaps in unsupervised transfer, cross-domain knowledge distillation, and interpretability, offering a roadmap for future work in this evolving field.

ABSTRACT

The application of transfer learning, leveraging knowledge from source domains to enhance model performance in a target domain, has significantly grown, supporting diverse real-world applications. Its success often relies on shared knowledge between domains, typically required in these methodologies. Commonly, methods assume identical feature and label spaces in both domains, known as homogeneous transfer learning. However, this is often impractical as source and target domains usually differ in these spaces, making precise data matching challenging and costly. Consequently, heterogeneous transfer learning (HTL), which addresses these disparities, has become a vital strategy in various tasks. In this paper, we offer an extensive review of over 60 HTL methods, covering both data-based and model-based approaches. We describe the key assumptions and algorithms of these methods and systematically categorize them into instance-based, feature representation-based, parameter regularization, and parameter tuning techniques. Additionally, we explore applications in natural language processing, computer vision, multimodal learning, and biomedicine, aiming to deepen understanding and stimulate further research in these areas. Our paper includes recent advancements in HTL, such as the introduction of transformer-based models and multimodal learning techniques, ensuring the review captures the latest developments in the field. We identify key limitations in current HTL studies and offer systematic guidance for future research, highlighting areas needing further exploration and suggesting potential directions for advancing the field.

Motivation & Objective

• To provide a comprehensive, up-to-date review of heterogeneous transfer learning (HTL) methods developed after 2017, addressing the limitations of prior surveys that predate major advancements.
• To analyze the challenges in HTL, including disparate feature spaces, label spaces, and data distributions, which hinder traditional homogeneous transfer learning.
• To examine the role of modern architectures like transformers (e.g., BERT, GPT) and large-scale pre-trained models in enabling effective knowledge transfer across heterogeneous domains.
• To identify critical research gaps in unsupervised HTL, cross-domain knowledge distillation, and model interpretability, especially in high-stakes applications.
• To guide future research by outlining promising directions, including multimodal learning, efficient fine-tuning, and robust, interpretable HTL frameworks.

Proposed method

• The survey adopts a systematic, taxonomy-driven approach to categorize HTL methods into data-based and model-based paradigms, emphasizing their mechanisms for handling domain heterogeneity.
• It reviews data-based methods that align source and target domains via feature space transformation, distribution alignment, or instance weighting, even when feature spaces differ.
• It examines model-based methods that transfer knowledge through architectural adaptation, such as shared representations, attention mechanisms, or cross-attention in multimodal settings.
• The paper analyzes the integration of pre-trained models—especially large language models—as foundational knowledge sources for HTL, enabling zero-shot or few-shot adaptation.
• It discusses multimodal HTL techniques that unify representations across modalities (e.g., text, image, audio) using contrastive learning or cross-attention to enable cross-domain transfer.
• The survey evaluates knowledge distillation in HTL, highlighting its limitations due to domain misalignment and the risk of negative transfer, while suggesting potential for novel cross-domain distillation techniques.

Experimental results

Research questions

• RQ1 How have recent developments in transformer-based models and large foundation models transformed the landscape of heterogeneous transfer learning since 2017?
• RQ2 What are the core methodological differences and limitations between data-based and model-based approaches in handling heterogeneous domains?
• RQ3 In what ways do multimodal learning and cross-attention mechanisms enhance knowledge transfer across diverse data modalities in HTL?
• RQ4 Why is knowledge distillation less effective in heterogeneous settings, and what novel techniques could overcome its limitations in cross-domain knowledge transfer?
• RQ5 What are the key challenges and opportunities in improving interpretability and reducing negative transfer in heterogeneous transfer learning systems?

Key findings

• The survey identifies that recent advances in HTL are increasingly driven by large pre-trained models, which provide robust, task-agnostic initialization and significantly reduce training costs through fine-tuning.
• Multimodal HTL has emerged as a powerful paradigm, especially in tasks like visual question answering and cross-modal retrieval, due to its ability to align diverse data types through attention mechanisms.
• Knowledge distillation remains ineffective in HTL when source and target domains differ in task or data structure, as it assumes aligned input-output spaces, risking negative transfer.
• Interpretability is a critical but underdeveloped area in HTL, as complex, non-intuitive interactions in transferred representations hinder model debugging and trust in high-stakes domains.
• Despite progress, unsupervised and weakly supervised HTL remain underexplored, representing a major frontier for future research to bridge the gap between abundant source data and scarce labeled target data.
• The survey concludes that while HTL is essential for real-world deployment due to data scarcity, scalable, robust, and interpretable frameworks are still needed for widespread adoption.