[Paper Review] Meta-Learning with MAML on Trees

Jezabel Garcia, Federica Freddi+6 more|arXiv (Cornell University)|Mar 8, 2021

Topic Modeling16 references4 citations

TL;DR

This paper proposes TreeMAML, a model-agnostic meta-learning algorithm that leverages hierarchical task structures—specifically, language phylogenetic trees—to improve few-shot cross-lingual natural language understanding. By pooling gradients across clusters in a tree-structured adaptation process, TreeMAML achieves a 3% accuracy gain over state-of-the-art methods on the XNLI benchmark, demonstrating improved transfer learning for under-resourced languages.

ABSTRACT

In meta-learning, the knowledge learned from previous tasks is transferred to new ones, but this transfer only works if tasks are related. Sharing information between unrelated tasks might hurt performance, and it is unclear how to transfer knowledge across tasks with a hierarchical structure. Our research extends a model agnostic meta-learning model, MAML, by exploiting hierarchical task relationships. Our algorithm, TreeMAML, adapts the model to each task with a few gradient steps, but the adaptation follows the hierarchical tree structure: in each step, gradients are pooled across tasks clusters, and subsequent steps follow down the tree. We also implement a clustering algorithm that generates the tasks tree without previous knowledge of the task structure, allowing us to make use of implicit relationships between the tasks. We show that the new algorithm, which we term TreeMAML, performs better than MAML when the task structure is hierarchical for synthetic experiments. To study the performance of the method in real-world data, we apply this method to Natural Language Understanding, we use our algorithm to finetune Language Models taking advantage of the language phylogenetic tree. We show that TreeMAML improves the state of the art results for cross-lingual Natural Language Inference. This result is useful, since most languages in the world are under-resourced and the improvement on cross-lingual transfer allows the internationalization of NLP models. This results open the window to use this algorithm in other real-world hierarchical datasets.

Motivation & Objective

• To address the limitation of MAML in transferring knowledge across unrelated or hierarchically structured tasks.
• To improve few-shot learning performance in low-resource NLP scenarios, especially for under-resourced languages.
• To develop a method that can dynamically learn task hierarchies without prior knowledge of task relationships.
• To enhance cross-lingual transfer in natural language understanding by incorporating linguistic phylogenetic structures into meta-learning.

Proposed method

• TreeMAML modifies MAML by introducing a hierarchical gradient pooling mechanism that follows a tree structure during model adaptation.
• At each adaptation step, gradients are aggregated across task clusters at the same level of the tree before descending to finer-grained tasks.
• A modified hierarchical clustering algorithm (based on OTD) is used to learn the task tree structure dynamically from task similarities without prior knowledge.
• The algorithm supports both fixed trees (e.g., language phylogenetic trees) and learned trees, enabling flexibility in real-world applications.
• The method is model-agnostic and compatible with any backbone network and loss function, making it broadly applicable.
• The approach uses gradient-based similarity measures to infer task relationships, inspired by parameter importance scores from prior work.

Experimental results

Research questions

• RQ1 Can hierarchical task structures improve few-shot meta-learning performance compared to standard MAML?
• RQ2 How does the performance of TreeMAML vary when using a fixed language phylogenetic tree versus a dynamically learned task hierarchy?
• RQ3 To what extent does incorporating linguistic relationships (e.g., language families) enhance cross-lingual transfer in NLP?
• RQ4 Can TreeMAML outperform state-of-the-art methods like XMAML on cross-lingual natural language inference?
• RQ5 Does the algorithm generalize better to under-resourced languages when the task hierarchy is learned from data rather than predefined?

Key findings

• TreeMAML outperforms standard MAML and baseline models on synthetic multidimensional regression tasks, reducing mean squared error by half.
• On the XNLI benchmark, TreeMAML achieves a 3% accuracy improvement over the previous state of the art (XMAML) in cross-lingual natural language inference.
• Fixed TreeMAML performs better than MAML across all languages, with the largest gains observed in low-resource languages.
• Learned TreeMAML performs comparably or better than fixed TreeMAML in most cases, suggesting it can discover useful task relationships beyond predefined phylogenetic trees.
• In some cases, such as Greek and Thai, performance is lower due to oversimplified or inaccurate tree structures, highlighting the sensitivity of the method to tree quality.
• The results indicate that TreeMAML can exploit non-phylogenetic similarities—such as genre or lexical borrowing—when learning the task hierarchy, enhancing transfer beyond traditional linguistic classifications.