[Paper Review] Asymmetric Loss For Multi-Label Classification

Emanuel Ben-Baruch, Tal Ridnik+5 more|arXiv (Cornell University)|Sep 29, 2020

Advanced Neural Network ApplicationsComputer Science40 references64 citations

TL;DR

The paper introduces Asymmetric Loss (ASL) for multi-label classification to address severe negative–positive imbalance and mislabeled data, combining asymmetric focusing and asymmetric probability shifting to achieve state-of-the-art mAP on major datasets without changing architectures. It also provides an adaptive scheme to dynamically adjust asymmetry during training.

ABSTRACT

In a typical multi-label setting, a picture contains on average few positive labels, and many negative ones. This positive-negative imbalance dominates the optimization process, and can lead to under-emphasizing gradients from positive labels during training, resulting in poor accuracy. In this paper, we introduce a novel asymmetric loss ("ASL"), which operates differently on positive and negative samples. The loss enables to dynamically down-weights and hard-thresholds easy negative samples, while also discarding possibly mislabeled samples. We demonstrate how ASL can balance the probabilities of different samples, and how this balancing is translated to better mAP scores. With ASL, we reach state-of-the-art results on multiple popular multi-label datasets: MS-COCO, Pascal-VOC, NUS-WIDE and Open Images. We also demonstrate ASL applicability for other tasks, such as single-label classification and object detection. ASL is effective, easy to implement, and does not increase the training time or complexity. Implementation is available at: https://github.com/Alibaba-MIIL/ASL.

Motivation & Objective

• Address the challenge of severe negative–positive label imbalance in multi-label image classification.
• Develop a loss function that down-weights easy negatives and discards very easy negatives, while preserving positive gradient signals.
• Design an adaptive mechanism to dynamically adjust asymmetry during training to simplify hyper-parameter tuning.
• Show that ASL achieves state-of-the-art results on major multi-label benchmarks without architectural changes.
• Demonstrate ASL's applicability to related tasks beyond multi-label classification.

Proposed method

• Define a binary cross-entropy/focal-loss framework with label-wise logits z_k and probabilities p = sigmoid(z).
• Introduce asymmetric focusing by decoupling positive and negative focusing parameters (gamma_plus and gamma_minus).
• Introduce asymmetric probability shifting where negative probabilities are shifted p_m = max(p - m, 0) to enable hard thresholding of easy negatives.
• Formulate ASL as L_plus = (1 - p)^{gamma_plus} log(p) and L_minus = (p_m)^{gamma_minus} log(1 - p_m).
• Optionally set gamma_plus = 0 to keep positive loss as standard CE while tuning gamma_minus for asymmetry.
• Provide gradient and probability analyses to show benefits, including rejection of mislabeled negatives and balanced sample contributions.
• Propose an adaptive asymmetry scheme that updates gamma_minus to target a desired probability gap Delta p via gamma_minus <- gamma_minus + lambda(Delta p - Delta p_target).
• Demonstrate that combining both asymmetries yields best performance compared to single-asymmetry or static weighting.

Experimental results

Research questions

• RQ1 Does an asymmetric loss addressing negative–positive imbalance improve multi-label classification performance without changing network architecture?
• RQ2 How do asymmetric focusing and probability shifting contribute to improved gradient flow and robustness to mislabeled negatives?
• RQ3 Can ASL dynamically adapt its asymmetry during training to simplify hyper-parameter tuning while maintaining or improving performance?
• RQ4 Is ASL effective across multiple major multi-label datasets and backbones, and does it extend to related tasks such as single-label classification and object detection?

Key findings

• ASL outperforms cross-entropy and focal loss on MS-COCO, Pascal-VOC, NUS-WIDE, and Open Images, achieving state-of-the-art results.
• On MS-COCO, ASL with ResNet101 reaches 85.0 mAP (vs 84.0 with CE and 85.1 with focal), and ASL with TResNet-L reaches 86.6 mAP (top-1).
• Combining asymmetric focusing and probability shifting yields the best MS-COCO results (86.6 mAP) versus using either mechanism alone.
• Adaptive asymmetry experiments show higher mAP (up to 86.4) with Delta p_target settings, though fixed ASL variants can achieve slightly higher scores in some cases.
• ASL improves performance across multiple backbones (OFA-595, ResNet101, TResNet-L) and benefits from better pretraining and higher input resolution (e.g., 448 to 640).
• ASL also shows benefits for related tasks such as single-label classification and object detection, indicating broader applicability.