[Paper Review] A Unified Perspective on Multi-Domain and Multi-Task Learning

Yongxin Yang, Timothy M. Hospedales|arXiv (Cornell University)|Dec 23, 2014

Domain Adaptation and Few-Shot Learning30 references92 citations

TL;DR

This paper proposes a unified neural network framework for multi-domain and multi-task learning (MDL/MTL) by introducing multivariate semantic descriptors to represent tasks and domains, enabling improved knowledge sharing. The method outperforms existing approaches in MTL, MDL, zero-shot learning (ZSL), and introduces a novel task: zero-shot domain adaptation (ZSDA), achieving 87.66% accuracy on AwA and 0.78 RMSE on the restaurant dataset.

ABSTRACT

In this paper, we provide a new neural-network based perspective on multi-task learning (MTL) and multi-domain learning (MDL). By introducing the concept of a semantic descriptor, this framework unifies MDL and MTL as well as encompassing various classic and recent MTL/MDL algorithms by interpreting them as different ways of constructing semantic descriptors. Our interpretation provides an alternative pipeline for zero-shot learning (ZSL), where a model for a novel class can be constructed without training data. Moreover, it leads to a new and practically relevant problem setting of zero-shot domain adaptation (ZSDA), which is the analogous to ZSL but for novel domains: A model for an unseen domain can be generated by its semantic descriptor. Experiments across this range of problems demonstrate that our framework outperforms a variety of alternatives.

Motivation & Objective

• To unify multi-domain and multi-task learning under a single neural network framework using semantic descriptors.
• To improve knowledge sharing by leveraging multi-variate task/domain metadata instead of treating tasks/domains as atomic categories.
• To enable zero-shot learning (ZSL) via semantic descriptor-based model synthesis.
• To introduce and address the novel problem of zero-shot domain adaptation (ZSDA), where models for unseen domains are generated from their semantic descriptors.
• To demonstrate superior performance across MTL, MDL, ZSL, and ZSDA benchmarks.

Proposed method

• The framework uses a multivariate semantic descriptor to represent tasks and domains, enabling structured information sharing beyond categorical indexing.
• It formulates model learning as a factorization problem: W = LS, where L represents shared latent predictors and S represents task/domain-specific coefficients derived from semantic descriptors.
• Semantic descriptors are concatenated from discrete variables (e.g., domain ID and task type) to form a distributed representation of tasks/domains.
• The method generalizes existing MTL algorithms (e.g., GO-MTL, RMTL) as special cases under different assumptions about the descriptor structure.
• For zero-shot learning, the framework synthesizes classifiers for unseen classes using only their semantic descriptors, bypassing training data.
• For zero-shot domain adaptation (ZSDA), it generates models for entirely unseen domains using only their semantic descriptors, a novel contribution.

Experimental results

Research questions

• RQ1 Can a unified framework unify multi-domain and multi-task learning by leveraging semantic descriptors?
• RQ2 How does using multivariate semantic descriptors improve knowledge sharing compared to atomic categorical indexing in MTL/MDL?
• RQ3 Can the framework enable zero-shot learning (ZSL) by synthesizing models for unseen classes using only their semantic descriptors?
• RQ4 Is zero-shot domain adaptation (ZSDA) a viable and meaningful problem setting, and can it be effectively solved using semantic descriptors?
• RQ5 Does the proposed method outperform existing MTL, MDL, ZSL, and ZSDA baselines across diverse benchmarks?

Key findings

• On the AwA dataset, the proposed method achieved 87.66% multi-class accuracy in MTL, outperforming standard STL (85.34%) and existing MTL methods like MTFL (87.09%) and GO-MTL (86.20%).
• In zero-shot learning (ZSL), the method achieved 43.79% multi-class accuracy, surpassing the DAP baseline of 41.03% using DeCAF features.
• For the restaurant & consumer dataset, the method achieved a test RMSE of 0.78, significantly outperforming the next-best method (GO-MTL at 1.06).
• The framework successfully enables zero-shot domain adaptation (ZSDA), synthesizing models for unseen domains using only their semantic descriptors, a novel contribution not previously addressed.
• The method generalizes existing MTL algorithms as special cases, demonstrating that they implicitly assume atomic task/domain indexing, whereas the framework leverages richer metadata for better performance.