[Paper Review] Large Loss Matters in Weakly Supervised Multi-Label Classification

Youngwook Kim, Jae Myung Kim+2 more|arXiv (Cornell University)|Jun 8, 2022

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes a novel large loss reweighting strategy for weakly supervised multi-label classification, where samples with large relative losses are prioritized during training to improve model generalization. By dynamically adjusting loss weighting based on relative loss magnitude within each batch, the method achieves state-of-the-art performance, with gains of up to +6.6 mAP on NUSWIDE and +0.4 mAP on OpenImages V3 compared to baseline methods.

ABSTRACT

Weakly supervised multi-label classification (WSML) task, which is to learn a multi-label classification using partially observed labels per image, is becoming increasingly important due to its huge annotation cost. In this work, we first regard unobserved labels as negative labels, casting the WSML task into noisy multi-label classification. From this point of view, we empirically observe that memorization effect, which was first discovered in a noisy multi-class setting, also occurs in a multi-label setting. That is, the model first learns the representation of clean labels, and then starts memorizing noisy labels. Based on this finding, we propose novel methods for WSML which reject or correct the large loss samples to prevent model from memorizing the noisy label. Without heavy and complex components, our proposed methods outperform previous state-of-the-art WSML methods on several partial label settings including Pascal VOC 2012, MS COCO, NUSWIDE, CUB, and OpenImages V3 datasets. Various analysis also show that our methodology actually works well, validating that treating large loss properly matters in a weakly supervised multi-label classification. Our code is available at https://github.com/snucml/LargeLossMatters.

Motivation & Objective

• To address the challenge of noisy and incomplete labels in multi-label image classification.
• To improve model generalization in weakly supervised learning by focusing on samples with large training losses.
• To develop a dynamic loss reweighting mechanism that adapts thresholding based on relative loss magnitude within each mini-batch.
• To evaluate the effectiveness of relative vs. absolute loss thresholds in loss reweighting strategies.
• To demonstrate improved class activation maps and model interpretability through visualization.

Proposed method

• The method introduces a relative loss thresholding mechanism, where the threshold $ R(t) = R_0 - t \cdot \Delta_{\text{rel}} $ is adjusted over time based on the relative magnitude of losses in each mini-batch.
• Three variants—LL-R, LL-Ct, and LL-Cp—are proposed, differing in how the threshold is applied across samples: random, class-wise, or per-sample reweighting.
• An absolute loss variant is also introduced, where $ R(t) = R_0 - t \cdot \Delta_{\text{abs}} $, using absolute loss values instead of relative ranking.
• Hyperparameters $ \Delta_{\text{rel}} $, $ \Delta_{\text{abs}} $, and $ R_0 $ are searched over discrete ranges to optimize performance.
• The model is trained end-to-end or with linear probing, using a 20% validation split from the training set, and the best model is selected based on mAP.
• Class Activation Maps (CAMs) are visualized to compare localization fidelity with baseline methods like Naive AN and ROLE.

Figure 1 : Class Activation Mapping results in COCO test images.

Experimental results

Research questions

• RQ1 Does prioritizing samples with large relative losses improve performance in weakly supervised multi-label classification?
• RQ2 How does relative loss thresholding compare to absolute loss thresholding in terms of mAP and robustness?
• RQ3 Can dynamic loss reweighting lead to better localization and explanation via Class Activation Maps?
• RQ4 What is the impact of different loss reweighting strategies (LL-R, LL-Ct, LL-Cp) on model generalization?
• RQ5 How do the proposed methods perform on both synthetic partial label datasets and real-world partial label data?

Key findings

• On the NUSWIDE dataset with artificially created partial labels, LL-Ct abs achieved a +6.6 mAP gain over Naive AN and a +0.3 mAP gain over LL-relative.
• On OpenImages V3 with real partial labels, the absolute variants (LL-R abs, LL-Ct abs, LL-Cp abs) achieved a +0.3 to +0.4 mAP improvement over Naive AN.
• The LL-Ct abs variant achieved 48.6 mAP on NUSWIDE, outperforming both Naive AN (42.0 mAP) and LL-relative (48.3 mAP).
• In the OpenImages V3 G1–G5 group evaluation, LL-Ct abs achieved 89.3 mAP on G5, compared to 88.8 mAP for Naive AN, showing consistent gains across groups.
• Visualizations of Class Activation Maps showed that LL-Ct produces more accurate localization of objects compared to Naive AN and ROLE, indicating better alignment with human interpretation.
• The absolute loss variant (LL-Ct abs) maintained strong performance across both synthetic and real-world datasets, demonstrating robustness to different loss criteria.