[论文解读] Long-Tailed Recognition via Weight Balancing

Shaden Alshammari, Yu-Xiong Wang+2 more|arXiv (Cornell University)|Mar 27, 2022

Domain Adaptation and Few-Shot Learning被引用 9

一句话总结

本文提出了一种简单但有效的两阶段训练方法，通过使用权重衰减和MaxNorm正则化来平衡分类器权重，从而在CIFAR100-LT（不平衡因子100）上实现了53.35%的最先进准确率，且无需复杂架构或数据增强。

ABSTRACT

In the real open world, data tends to follow long-tailed class distributions, motivating the well-studied long-tailed recognition (LTR) problem. Naive training produces models that are biased toward common classes in terms of higher accuracy. The key to addressing LTR is to balance various aspects including data distribution, training losses, and gradients in learning. We explore an orthogonal direction, weight balancing, motivated by the empirical observation that the naively trained classifier has "artificially" larger weights in norm for common classes (because there exists abundant data to train them, unlike the rare classes). We investigate three techniques to balance weights, L2-normalization, weight decay, and MaxNorm. We first point out that L2-normalization "perfectly" balances per-class weights to be unit norm, but such a hard constraint might prevent classes from learning better classifiers. In contrast, weight decay penalizes larger weights more heavily and so learns small balanced weights; the MaxNorm constraint encourages growing small weights within a norm ball but caps all the weights by the radius. Our extensive study shows that both help learn balanced weights and greatly improve the LTR accuracy. Surprisingly, weight decay, although underexplored in LTR, significantly improves over prior work. Therefore, we adopt a two-stage training paradigm and propose a simple approach to LTR: (1) learning features using the cross-entropy loss by tuning weight decay, and (2) learning classifiers using class-balanced loss by tuning weight decay and MaxNorm. Our approach achieves the state-of-the-art accuracy on five standard benchmarks, serving as a future baseline for long-tailed recognition.

研究动机与目标

• 解决深度学习模型在长尾数据分布中对常见类别存在的偏差问题。
• 探究性能差异的根本原因：由于训练数据丰富，导致常见类别分类器权重范数人为过大。
• 探索正交正则化技术，以平衡各类别的权重范数，提升罕见类别性能。
• 提出一种简单有效的两阶段训练范式，将特征学习与分类器学习解耦，以提升泛化能力。

提出的方法

• 对所有类别分类器权重施加L2归一化以强制其单位范数，但发现该方法对最优性能而言过于严格。
• 使用权重衰减对大权重施加更强惩罚，鼓励各类别权重范数更小且更均衡。
• 实施MaxNorm约束，将权重范数限制在固定半径内，允许小权重增长，同时防止过度优化。
• 采用两阶段训练策略：首先使用交叉熵损失和调优后的权重衰减训练特征，然后使用类别平衡损失及组合权重衰减与MaxNorm对分类器进行微调。
• 通过二维预 logits 层可视化训练过程中权重的演化，以展示范数平衡的动力学过程。
• 在CIFAR100-LT上进行广泛的消融实验，评估每种正则化技术及训练阶段的影响。

Figure 3 : Weight decay helps learn balanced weights at hidden layers . We compare the norm distribution at each layer (which has 512 filters) from the naive model ( orange ) and the one trained with weight decay ( blue ). For each layer of a model, we sort the filter weights of each layer from high

实验结果

研究问题

• RQ1 为何朴素训练的模型在长尾数据集的罕见类别上表现不佳？
• RQ2 与现有的数据重加权或损失重加权方法相比，权重范数平衡能在多大程度上提升长尾识别准确率？
• RQ3 不同的正则化技术（L2归一化、权重衰减、MaxNorm）如何影响分类器权重的平衡性与大小？
• RQ4 何种最优训练策略（如两阶段微调）能有效利用权重平衡提升长尾识别性能？
• RQ5 在不改变网络架构或数据增强的前提下，简单且广为人知的正则化方法能否超越复杂的最先进方法？

主要发现

• 仅使用权重衰减，CIFAR100-LT（IF=100）上的准确率即可从朴素训练的38.38%提升至46.08%，相对提升+8%。
• 使用类别平衡损失与权重衰减对最后一层进行微调，准确率提升至52.42%，表明损失与正则化结合具有显著优势。
• 在最佳设置（CB + WD）基础上增加MaxNorm，额外提升1%，在CIFAR100-LT上达到53.35%的top-1准确率。
• 对最后两层进行微调（CB + WD + MaxNorm）可达到53.55%的准确率，略高于仅微调最后一层的设置。
• 过度微调更多层会降低性能，可能是因为类别平衡损失在特征学习阶段引入了梯度冲突。
• 可视化结果证实，权重衰减与MaxNorm能显著提升各类别权重范数的平衡性，尤其有效降低了常见类别权重的主导性。

Figure 4 : Tuning weight decay drastically improves long-tailed recognition performance. We do not use any class-balancing techniques but simply use CE loss and tune weight decay $\lambda$ to regularize all network weight. For example, tuning $\lambda$ yields $46.1\%$ accuracy on CIFAR100-LT (IF=100