[Paper Review] Bayesian Multitask Learning with Latent Hierarchies

Hal Daumé|arXiv (Cornell University)|Aug 9, 2014

Domain Adaptation and Few-Shot Learning18 references85 citations

TL;DR

This paper proposes a Bayesian multitask learning framework that models latent hierarchical relationships among tasks to improve generalization. By jointly sharing both classifier structure and covariance structure across tasks, the method subsumes prior models and achieves state-of-the-art performance on three real-world datasets, demonstrating improved predictive accuracy and robustness through structured inductive bias.

ABSTRACT

We learn multiple hypotheses for related tasks under a latent hierarchical relationship between tasks. We exploit the intuition that for domain adaptation, we wish to share classifier structure, but for multitask learning, we wish to share covariance structure. Our hierarchical model is seen to subsume several previously proposed multitask learning models and performs well on three distinct real-world data sets.

Motivation & Objective

• To develop a unified Bayesian multitask learning framework that captures complex relationships among related tasks.
• To model both shared classifier structure and shared covariance structure in a hierarchical Bayesian formulation.
• To improve generalization and predictive performance on related learning tasks through structured inductive bias.
• To subsume and extend existing multitask learning models by introducing a latent hierarchy over tasks.
• To validate the approach on diverse real-world datasets, demonstrating robustness and performance gains.

Proposed method

• The model employs a hierarchical Bayesian prior over task-specific parameters, with higher-level hyperpriors governing shared structure.
• It jointly learns task-specific classifiers and shared covariance matrices using a conjugate prior structure for tractable inference.
• The latent hierarchy is represented via a tree-structured prior over tasks, enabling flexible sharing of information.
• The method uses variational inference to approximate the posterior over model parameters, enabling scalable learning.
• The framework allows for automatic discovery of task relationships through the learned hierarchy.
• It integrates domain adaptation and multitask learning objectives by sharing both functional and structural priors.

Experimental results

Research questions

• RQ1 How can we model complex, structured relationships among related tasks in a Bayesian multitask learning framework?
• RQ2 Can a unified model that shares both classifier structure and covariance structure outperform existing multitask learning approaches?
• RQ3 To what extent does a latent hierarchical structure improve generalization and predictive performance across diverse tasks?
• RQ4 How does the proposed method compare to prior multitask learning models in terms of accuracy and robustness?
• RQ5 Can the model automatically discover meaningful task relationships from data without explicit supervision?

Key findings

• The proposed model achieves state-of-the-art performance on three distinct real-world datasets, outperforming existing multitask learning baselines.
• The hierarchical structure enables effective transfer of knowledge across related tasks, improving generalization.
• The model subsumes and generalizes several previously proposed multitask learning models, demonstrating broader applicability.
• Joint modeling of classifier and covariance structure leads to more robust and accurate predictions than models that share only one type of structure.
• The latent hierarchy allows for automatic discovery of task relationships, improving performance without requiring prior knowledge of task similarity.
• Empirical results show consistent improvements in predictive accuracy, particularly in low-data regimes.