[Paper Review] Structured Prediction for Conditional Meta-Learning

Ruohan Wang, Yiannis Demiris+1 more|arXiv (Cornell University)|Feb 20, 2020

Domain Adaptation and Few-Shot Learning49 references4 citations

TL;DR

This paper proposes Task-adaptive Structured Meta-Learning (TASML), a model-agnostic, non-parametric framework that uses structured prediction to learn task-specific meta-parameters by weighting meta-training tasks based on their similarity to the target task. TASML improves few-shot learning performance on mini-ImageNet and tiered-ImageNet, outperforming state-of-the-art methods while maintaining computational efficiency.

ABSTRACT

The goal of optimization-based meta-learning is to find a single initialization shared across a distribution of tasks to speed up the process of learning new tasks. Conditional meta-learning seeks task-specific initialization to better capture complex task distributions and improve performance. However, many existing conditional methods are difficult to generalize and lack theoretical guarantees. In this work, we propose a new perspective on conditional meta-learning via structured prediction. We derive task-adaptive structured meta-learning (TASML), a principled framework that yields task-specific objective functions by weighing meta-training data on target tasks. Our non-parametric approach is model-agnostic and can be combined with existing meta-learning methods to achieve conditioning. Empirically, we show that TASML improves the performance of existing meta-learning models, and outperforms the state-of-the-art on benchmark datasets.

Motivation & Objective

• To address the limitations of unconditional meta-learning in handling complex, multi-modal task distributions by enabling task-specific meta-parameters.
• To provide a theoretically grounded, non-parametric framework for conditional meta-learning that generalizes across diverse meta-learning methods.
• To improve generalization and performance in few-shot learning by dynamically recalling the most relevant past tasks for each new task.
• To balance computational efficiency and accuracy in conditional meta-learning through an efficient optimization algorithm.
• To empirically validate the framework on standard few-shot classification benchmarks, demonstrating superiority over existing state-of-the-art methods.

Proposed method

• TASML frames the inner optimization process as a structured prediction problem, where the task-specific meta-parameters are predicted based on the target task's features.
• It uses a weighted combination of meta-training tasks, with weights derived from a structured prediction model that measures similarity between the target task and past tasks.
• The framework employs a non-parametric approach that requires access to meta-training data at test time to compute task-specific objectives.
• An efficient algorithm is introduced to mitigate the computational cost of optimizing task-specific objectives by filtering and prioritizing the most relevant training tasks.
• The method is model-agnostic and can be combined with any existing meta-learning algorithm to enable conditional adaptation.
• A structured prediction objective is formulated using a kernelized similarity measure (e.g., Gaussian kernel) to compute attention weights over meta-training tasks.

Experimental results

Research questions

• RQ1 Can structured prediction provide a principled and generalizable framework for conditional meta-learning?
• RQ2 How does task-specific meta-optimization based on weighted past tasks improve few-shot generalization compared to shared initialization?
• RQ3 What is the impact of filtering the most relevant meta-training tasks on performance and efficiency?
• RQ4 How do different kernel functions affect the robustness and accuracy of the structured prediction component?
• RQ5 Can the proposed method achieve state-of-the-art performance while remaining model-agnostic and efficient?

Key findings

• TASML achieves state-of-the-art performance on both mini-ImageNet and tiered-ImageNet, with 78.51% accuracy on mini-ImageNet and 83.03% on tiered-ImageNet under 5-way-5-shot settings.
• The method significantly improves the accuracy of existing meta-learning models by adapting them into conditional variants, demonstrating strong generalization across diverse tasks.
• Performance remains robust across different values of M (number of top tasks used), with minimal performance drop even at M=100, indicating effective filtering of irrelevant tasks.
• The inclusion of both target-task and related-task objectives is essential: excluding either leads to overfitting or underfitting, with optimal performance when both are weighted equally.
• The Gaussian kernel outperforms linear and Laplace kernels in terms of accuracy and robustness to hyperparameter tuning, particularly in bandwidth sensitivity.
• The method maintains computational efficiency despite requiring access to meta-training data at test time, thanks to an efficient filtering and optimization strategy.