[论文解读] A Unified Perspective for Loss-Oriented Imbalanced Learning via Localization

Zitai Wang, Qianqian Xu+5 more|arXiv (Cornell University)|Oct 7, 2023

Imbalanced Data Classification TechniquesComputer Science被引用 3

一句话总结

该论文提出了一种基于新型数据相关收缩技术与局部Lipschitz连续性的损失导向类不平衡学习统一泛化分析，揭示了重加权与logit校正如何协同提升少数类泛化性能。该方法实现了一种有理论依据的学习算法，使重加权与泛化界对齐，并在延迟重加权阶段移除乘法logit校正项，从而在ImageNet-LT和iNaturalist数据集上取得最先进性能。

ABSTRACT

Due to the inherent imbalance in real-world datasets, naïve Empirical Risk Minimization (ERM) tends to bias the learning process towards the majority classes, hindering generalization to minority classes. To rebalance the learning process, one straightforward yet effective approach is to modify the loss function via class-dependent terms, such as re-weighting and logit-adjustment. However, existing analysis of these loss-oriented methods remains coarse-grained and fragmented, failing to explain some empirical results. After reviewing prior work, we find that the properties used through their analysis are typically global, i.e., defined over the whole dataset. Hence, these properties fail to effectively capture how class-dependent terms influence the learning process. To bridge this gap, we turn to explore the localized versions of such properties i.e., defined within each class. Specifically, we employ localized calibration to provide consistency validation across a broader range of losses and localized Lipschitz continuity to provide a fine-grained generalization bound. In this way, we reach a unified perspective for improving and adjusting loss-oriented methods. Finally, a principled learning algorithm is developed based on these insights. Empirical results on both traditional ResNets and foundation models validate our theoretical analyses and demonstrate the effectiveness of the proposed method.

研究动机与目标

• 为类不平衡学习中损失修改方法的零散且粗粒度的理论理解提供解决方案。
• 在细粒度泛化界下统一分析重加权与logit校正技术。
• 解释现有理论无法捕捉的实证现象——如延迟重加权（DRW）的成功以及DRW与乘法logit项之间的不相容性。
• 基于理论洞见开发一种有原则的学习算法，以提升少数类的泛化性能。

提出的方法

• 引入局部Lipschitz连续性扩展，以捕捉类别特定的损失行为，替代全局Lipschitz假设。
• 提出一种数据相关收缩技术，推导出对类别不平衡敏感的细粒度泛化界。
• 将该泛化界应用于向量缩放（VS）损失，揭示加法项（重加权）与乘法项（logit校正）的独立作用。
• 设计一种有原则的学习算法，使重加权与泛化界对齐，并在DRW阶段移除乘法logit项，以避免不相容性。
• 将该方法与SAM（Sharpness-Aware Minimization）集成，以提升鲁棒性与泛化性能。

实验结果

研究问题

• RQ1 重加权与logit校正机制如何共同影响类不平衡学习中的泛化性能？
• RQ2 尽管理论存在空白，为何延迟重加权（DRW）优于标准重加权？
• RQ3 为何DRW与乘法logit校正（如VS损失中）的组合在实证中表现次优？
• RQ4 能否通过单一泛化界统一解释重加权与logit校正？

主要发现

• 所提方法在iNaturalist数据集上达到64.7%的平衡准确率，优于此前最先进方法（DisAlign为70.6%）
• 在ImageNet-LT上，该方法达到54.1%的平衡准确率，超过最佳基线方法（DRO-LT为53.5%）
• TLA（训练损失对齐）方案使VS+TLA+ADRW在高不平衡度（ρ=100）下优于VS与VS+DRW
• 该方法解释了当ρ=10时，CE+ADRW在性能上优于LDAM+ADRW的反直觉现象，归因于仅靠重加权即可重新平衡泛化界
• 实证结果证实，在DRW阶段移除乘法logit项可提升性能，验证了关于不相容性的理论洞察