[Paper Review] Learning Prototype-oriented Set Representations for Meta-Learning

Dandan Guo, Long Tian+3 more|arXiv (Cornell University)|Oct 18, 2021

Domain Adaptation and Few-Shot Learning40 references4 citations

TL;DR

This paper proposes a prototype-oriented optimal transport (POT) framework to improve summary networks for set-structured data in meta-learning by modeling each set as a distribution over global prototypes and minimizing the regularized optimal transport distance between the empirical set distribution and the prototype-based distribution. The method enhances feature representation learning in few-shot classification and generative modeling with state-of-the-art performance on MNIST, Flowers, and Animal Faces benchmarks.

ABSTRACT

Learning from set-structured data is a fundamental problem that has recently attracted increasing attention, where a series of summary networks are introduced to deal with the set input. In fact, many meta-learning problems can be treated as set-input tasks. Most existing summary networks aim to design different architectures for the input set in order to enforce permutation invariance. However, scant attention has been paid to the common cases where different sets in a meta-distribution are closely related and share certain statistical properties. Viewing each set as a distribution over a set of global prototypes, this paper provides a novel prototype-oriented optimal transport (POT) framework to improve existing summary networks. To learn the distribution over the global prototypes, we minimize its regularized optimal transport distance to the set empirical distribution over data points, providing a natural unsupervised way to improve the summary network. Since our plug-and-play framework can be applied to many meta-learning problems, we further instantiate it to the cases of few-shot classification and implicit meta generative modeling. Extensive experiments demonstrate that our framework significantly improves the existing summary networks on learning more powerful summary statistics from sets and can be successfully integrated into metric-based few-shot classification and generative modeling applications, providing a promising tool for addressing set-input and meta-learning problems.

Motivation & Objective

• Address the limitation of existing summary networks in capturing meaningful, transferable summary statistics from sets, especially under low-shot or meta-learning settings.
• Improve the representational power of set-encoding networks by introducing a global prototype-based distributional modeling approach.
• Provide a plug-and-play, unsupervised regularization mechanism via optimal transport to enhance model generalization without task-specific retraining.
• Enable better adaptation to unseen sets and distributions by learning a shared, structured representation across related tasks.
• Extend the applicability of summary networks to both supervised few-shot classification and unsupervised implicit generative modeling.

Proposed method

• Model each input set as an empirical distribution over its data points and a prototype-based distribution over K global, learnable prototypes.
• Use a summary network (e.g., Set Transformer, DeepSets) to encode the input set into a prototype proportion vector h_j, representing the importance of each global prototype.
• Formulate the learning objective as minimizing the regularized optimal transport (OT) distance between the empirical distribution P_j and the prototype-based distribution Q_j.
• Jointly optimize the task-specific loss (e.g., cross-entropy for classification, GAN loss for generation) and the prototype-oriented OT loss in an end-to-end manner.
• Introduce a differentiable, unsupervised regularization term that encourages the summary network to learn more robust and transferable representations.
• Apply the framework to both metric-based few-shot classification and implicit meta-generative modeling, with minimal architectural changes.

Experimental results

Research questions

• RQ1 Can a prototype-based distributional modeling approach improve the representational capacity of summary networks for set-structured inputs?
• RQ2 How does incorporating optimal transport between empirical and prototype-based distributions enhance generalization in few-shot learning?
• RQ3 To what extent does the proposed POT framework improve performance in implicit generative modeling with limited data?
• RQ4 Can the framework be applied as a plug-in module to existing meta-learning models without architectural overhaul?
• RQ5 Does the unsupervised OT regularization lead to better generalization on unseen distributions compared to supervised baselines?

Key findings

• The POT framework significantly improves baseline summary networks on few-shot image classification, achieving state-of-the-art performance on Omniglot and mini-ImageNet.
• On the rotated MNIST benchmark, CGAN+POT reduced FID from 213.30 to 159.51 at A = -180°, demonstrating superior generalization to unseen rotation angles.
• For few-shot image generation on Flowers and Animal Faces, DAGAN+POT achieved the lowest FID scores, indicating more realistic and diverse sample generation.
• The POT loss enables better zero-shot generalization, with DAGAN+POT outperforming DAGAN by 17.1% in FID on unseen Animal Faces categories.
• The method achieves consistent improvements across multiple benchmarks and tasks, validating its effectiveness as a plug-and-play regularizer.
• The framework successfully captures salient statistical characteristics of sets, even under low-shot and distribution-shift conditions, due to the structured prototype representation.