[Paper Review] Prototypical Networks for Multi-Label Learning

Zhuo Yang, Yufei Han+3 more|arXiv (Cornell University)|Nov 17, 2019

Text and Document Classification Technologies32 references4 citations

TL;DR

This paper proposes Prototypical Networks for Multi-Label Learning (PNML), a novel method that models multi-label classification by estimating class distributions in a shared non-linear embedding space, where each label's positive and negative instances form compact, separated components. The approach uses learned prototypes and Bregman divergence-based distance metrics to capture non-linear label dependencies, achieving state-of-the-art performance across 15 benchmark datasets with both PNML-single and PNML-multiple variants outperforming all baselines in 73% of evaluation cases.

ABSTRACT

We propose to formulate multi-label learning as a estimation of class distribution in a non-linear embedding space, where for each label, its positive data embeddings and negative data embeddings distribute compactly to form a positive component and negative component respectively, while the positive component and negative component are pushed away from each other. Duo to the shared embedding space for all labels, the distribution of embeddings preserves instances' label membership and feature matrix, thus encodes the feature-label relation and nonlinear label dependency. Labels of a given instance are inferred in the embedding space by measuring the probabilities of its belongingness to the positive or negative components of each label. Specially, the probabilities are modeled as the distance from the given instance to representative positive or negative prototypes. Extensive experiments validate that the proposed solution can provide distinctively more accurate multi-label classification than other state-of-the-art algorithms.

Motivation & Objective

• To address the challenge of modeling complex, non-linear label dependencies in multi-label learning.
• To jointly encode feature-label relationships and label correlations through a shared embedding space.
• To improve classification accuracy by estimating the distribution of positive and negative instances per label in a non-linear space.
• To develop a flexible framework that supports both single and multiple prototypes per label for trade-offs between accuracy and efficiency.

Proposed method

• The method maps all training instances into a shared non-linear embedding space using a shared feature encoder, preserving both label membership and feature structure.
• For each label, positive and negative instance embeddings are modeled as mixture components, with prototypes representing cluster centers in the embedding space.
• PNML-multiple learns multiple prototypes per label via adaptive clustering, while PNML-single uses a single prototype per component (mean of positive/negative embeddings).
• A label-specific Bregman divergence distance metric is jointly learned to measure instance proximity to positive and negative prototypes, enabling flexible distribution modeling.
• A label correlation regularizer is incorporated into the objective to further refine prototypes by exploiting inter-label dependencies.
• The model is trained end-to-end using a cross-entropy loss over predicted probabilities derived from distances to prototypes.

Experimental results

Research questions

• RQ1 Can modeling label distributions in a shared non-linear embedding space improve multi-label classification beyond linear or independent label modeling?
• RQ2 How effective is the use of learned prototypes and Bregman divergence metrics in capturing non-linear label dependencies?
• RQ3 Does joint optimization of embedding functions and label-specific distance metrics lead to better generalization than independent label learning?
• RQ4 To what extent do multiple prototypes per label enhance distribution modeling compared to a single prototype?
• RQ5 How does the proposed method perform across diverse real-world datasets with varying label complexity and sparsity?

Key findings

• PNML-single and PNML-multiple both outperform all 13 baselines in 73% of the 75 evaluation cases (55 out of 75), with statistically significant superiority confirmed by sign tests.
• PNML-multiple achieves higher performance than PNML-single in most cases, indicating that multiple prototypes better capture complex distribution patterns.
• PNML-single shows superior performance on Macro-F1, indicating better handling of rare or underrepresented labels compared to baselines.
• The ablation study confirms that the shared embedding function and learned distance metrics are essential, as PNML-I (independent labels) and PNML-D (Euclidean distance) perform significantly worse.
• Performance remains stable across different sampling rates, while runtime decreases with lower sampling rates, demonstrating computational efficiency with minimal accuracy loss.
• The label correlation regularizer contributes to improved prototype quality, enhancing overall classification performance.