[Paper Review] Label Structure Preserving Contrastive Embedding for Multi-Label Learning with Missing Labels

Zhongchen Ma, Lisha Li+2 more|arXiv (Cornell University)|Sep 3, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes CLML, a novel contrastive loss for multi-label image classification under missing labels, which integrates label correction and low-rank structure preservation to improve feature representation. By accurately identifying positive and negative instances per label and preserving global and local label dependencies, CLML boosts ResNet101 performance by 1.2–1.3% on MSCOCO, VOC, and NUS-WIDE.

ABSTRACT

Contrastive learning (CL) has shown impressive advances in image representation learning in whichever supervised multi-class classification or unsupervised learning. However, these CL methods fail to be directly adapted to multi-label image classification due to the difficulty in defining the positive and negative instances to contrast a given anchor image in multi-label scenario, let the label missing one alone, implying that borrowing a commonly-used way from contrastive multi-class learning to define them will incur a lot of false negative instances unfavorable for learning. In this paper, with the introduction of a label correction mechanism to identify missing labels, we first elegantly generate positives and negatives for individual semantic labels of an anchor image, then define a unique contrastive loss for multi-label image classification with missing labels (CLML), the loss is able to accurately bring images close to their true positive images and false negative images, far away from their true negative images. Different from existing multi-label CL losses, CLML also preserves low-rank global and local label dependencies in the latent representation space where such dependencies have been shown to be helpful in dealing with missing labels. To the best of our knowledge, this is the first general multi-label CL loss in the missing-label scenario and thus can seamlessly be paired with those losses of any existing multi-label learning methods just via a single hyperparameter. The proposed strategy has been shown to improve the classification performance of the Resnet101 model by margins of 1.2%, 1.6%, and 1.3% respectively on three standard datasets, MSCOCO, VOC, and NUS-WIDE. Code is available at https://github.com/chuangua/ContrastiveLossMLML.

Motivation & Objective

• Address the challenge of training robust multi-label models when labels are incomplete or missing during training.
• Overcome the limitation of standard contrastive learning in multi-label settings, where false negatives arise due to missing labels.
• Preserve low-rank global and local label dependencies in the embedding space to enhance representation learning under label scarcity.
• Develop a general-purpose contrastive loss that seamlessly integrates with existing multi-label learning frameworks via a single hyperparameter.

Proposed method

• Introduce a label correction mechanism to infer missing labels during training, reducing false negatives in contrastive learning.
• Define instance-level positive and negative pairs for each semantic label independently, improving contrastive signal accuracy.
• Propose a unique contrastive loss (CLML) that pulls true positives closer and pushes true negatives farther in the embedding space.
• Incorporate low-rank constraints on both global and local label structures to preserve semantic relationships among labels.
• Combine CLML with standard cross-entropy loss (BCE) to jointly optimize classification and contrastive representation learning.
• Use t-SNE visualization and ablation studies to validate the effectiveness of each component in the loss function.

Experimental results

Research questions

• RQ1 How can contrastive learning be effectively adapted to multi-label image classification when labels are missing?
• RQ2 To what extent can a label correction mechanism reduce false negative instances in contrastive learning under label incompleteness?
• RQ3 Can preserving low-rank global and local label dependencies improve representation learning in the presence of missing labels?
• RQ4 How does CLML compare to standard cross-entropy and other contrastive losses in terms of performance and robustness on standard benchmarks?

Key findings

• CLML improves the mAP of ResNet101 by 1.2%, 1.6%, and 1.3% on MSCOCO, VOC, and NUS-WIDE, respectively, outperforming baseline methods.
• The ablation study shows that CLML achieves the best performance across all evaluation metrics, with the highest mAP (87.2) and F1-score (80.3) on VOC.
• t-SNE visualization confirms that CLML enhances intra-class compactness and inter-class separation, especially for false negative samples.
• The label correction mechanism successfully detects and incorporates missing labels, as shown by looser feature distributions in the embedding space.
• Low-rank constraints significantly improve clustering quality, with tighter clusters and clearer boundaries in t-SNE plots.
• CLML reduces the distance between false negative samples and true positive clusters, demonstrating its ability to mitigate the impact of missing labels.