[Paper Review] Recent Advances in Transfer Learning for Cross-Dataset Visual Recognition: A Problem-Oriented Perspective

Jing Zhang, Wanqing Li+2 more|arXiv (Cornell University)|May 11, 2017

Domain Adaptation and Few-Shot LearningComputer Science281 references20 citations

TL;DR

This paper introduces a problem-oriented taxonomy of 17 cross-dataset visual recognition challenges based on data and label attributes, systematically reviewing transfer learning methods—both shallow and deep—for each. It identifies eight underexplored problems, offering practitioners a structured reference to match real-world tasks with suitable transfer learning solutions, while assessing dataset suitability and highlighting future research directions in transfer learning.

ABSTRACT

This paper takes a problem-oriented perspective and presents a comprehensive review of transfer learning methods, both shallow and deep, for cross-dataset visual recognition. Specifically, it categorises the cross-dataset recognition into seventeen problems based on a set of carefully chosen data and label attributes. Such a problem-oriented taxonomy has allowed us to examine how different transfer learning approaches tackle each problem and how well each problem has been researched to date. The comprehensive problem-oriented review of the advances in transfer learning with respect to the problem has not only revealed the challenges in transfer learning for visual recognition, but also the problems (e.g. eight of the seventeen problems) that have been scarcely studied. This survey not only presents an up-to-date technical review for researchers, but also a systematic approach and a reference for a machine learning practitioner to categorise a real problem and to look up for a possible solution accordingly.

Motivation & Objective

• To address the gap in existing survey papers that are method-driven rather than problem-driven, by proposing a systematic, problem-oriented taxonomy for cross-dataset visual recognition.
• To categorize cross-dataset visual recognition into 17 distinct problems based on data and label attributes, enabling targeted method selection for real-world applications.
• To evaluate the suitability of widely used datasets for each of the 17 problems, identifying limitations in current benchmarking practices.
• To reveal under-researched challenges—particularly eight of the seventeen problems—with minimal prior work, guiding future research.
• To provide researchers and practitioners with a practical, reference-ready framework to map real-world problems to appropriate transfer learning techniques.

Proposed method

• The paper defines a set of data and label attributes (e.g., domain shift, label space heterogeneity, data modality differences) to systematically categorize cross-dataset visual recognition into 17 distinct problems.
• It organizes the 17 problems into four scenarios: homogeneous feature and label spaces, heterogeneous feature spaces, heterogeneous label spaces, and heterogeneous feature and label spaces.
• For each problem, the paper reviews and analyzes existing shallow and deep transfer learning methods, including domain adaptation, domain generalization, and zero-shot learning techniques.
• The authors assess the suitability of major benchmark datasets (e.g., ImageNet, COCO, WebVision) for evaluating algorithms across each of the 17 problems, highlighting mismatches in data scale, modality, and label structure.
• The methodology includes a comparative analysis of how different approaches address each problem, based on formal definitions and task-oriented categorization from prior literature.
• The framework enables practitioners to map a real-world problem to a specific problem class and identify candidate solutions from the literature.

Experimental results

Research questions

• RQ1 Which transfer learning methods are most effective for each of the 17 problem classes in cross-dataset visual recognition?
• RQ2 How well-studied are each of the 17 identified problems, and which ones remain under-researched?
• RQ3 To what extent do current benchmark datasets support evaluation across the full spectrum of cross-dataset visual recognition problems?
• RQ4 How can practitioners systematically map a real-world visual recognition task to an appropriate transfer learning solution using this taxonomy?
• RQ5 What are the key challenges and future research directions in transfer learning for visual recognition, particularly for the under-explored problems?

Key findings

• Eight of the seventeen identified problems in cross-dataset visual recognition have received minimal research attention, indicating significant gaps in current literature.
• The paper reveals that existing survey papers often overlook problems involving heterogeneous label spaces, cross-modal recognition, and weakly labeled data, despite their practical relevance.
• Many widely used datasets, such as ImageNet and COCO, are ill-suited for evaluating online or continual transfer learning, as they lack dynamic or sequential data settings.
• Cross-modal transfer learning—e.g., from RGB to depth images—remains underdeveloped, despite its practical value in reducing data collection and annotation costs.
• The problem-oriented taxonomy enables practitioners to efficiently map real-world tasks to appropriate transfer learning methods, improving solution selection and reducing trial-and-error experimentation.
• The survey identifies that large-scale, versatile datasets suitable for evaluating diverse transfer learning problems—especially in online and continual learning settings—are largely missing, limiting algorithm evaluation and advancement.