[论文解读] Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models

Beier Zhu, Kaihua Tang+2 more|arXiv (Cornell University)|Oct 12, 2023

Domain Adaptation and Few-Shot LearningComputer Science被引用 3

一句话总结

本文提出广义对数调整（GLA），一种后处理方法，通过仅从下游数据估计预训练阶段的标签分布，从而消除CLIP等基础模型中的标签偏差，然后将去偏的零样本模型与微调模型进行集成。GLA在ImageNet上实现1.5个百分点的准确率提升，并在少样本和长尾基准测试中持续改进，其性能优于标准微调和集成方法，且在类别平衡分布下达到贝叶斯最优。

ABSTRACT

Foundation models like CLIP allow zero-shot transfer on various tasks without additional training data. Yet, the zero-shot performance is less competitive than a fully supervised one. Thus, to enhance the performance, fine-tuning and ensembling are also commonly adopted to better fit the downstream tasks. However, we argue that such prior work has overlooked the inherent biases in foundation models. Due to the highly imbalanced Web-scale training set, these foundation models are inevitably skewed toward frequent semantics, and thus the subsequent fine-tuning or ensembling is still biased. In this study, we systematically examine the biases in foundation models and demonstrate the efficacy of our proposed Generalized Logit Adjustment (GLA) method. Note that bias estimation in foundation models is challenging, as most pre-train data cannot be explicitly accessed like in traditional long-tailed classification tasks. To this end, GLA has an optimization-based bias estimation approach for debiasing foundation models. As our work resolves a fundamental flaw in the pre-training, the proposed GLA demonstrates significant improvements across a diverse range of tasks: it achieves 1.5 pp accuracy gains on ImageNet, an large average improvement (1.4-4.6 pp) on 11 few-shot datasets, 2.4 pp gains on long-tailed classification. Codes are in https://github.com/BeierZhu/GLA.

研究动机与目标

• 识别并解决由网络规模预训练数据分布偏斜引起的、被忽视的基础模型标签偏差问题。
• 开发一种实用方法，无需访问原始预训练数据集，即可对零样本基础模型进行去偏处理。
• 通过统计上最优的方式将去偏的零样本模型与微调模型进行集成，以提升下游性能。
• 在多种设置下评估该方法：多样本、少样本和长尾学习，涵盖多种微调范式。

提出的方法

• GLA通过一个约束优化问题估计基础模型中的标签偏差，最小化在下游数据集上的误差，且无需访问预训练数据。
• 它利用下游标签和模型预测结果，调整零样本模型的logit边界，以校正预训练分布偏斜。
• 该方法基于贝叶斯决策理论，使最终的集成分类器在类别平衡的目标分布下达到贝叶斯最优。
• GLA通过后处理方式对零样本模型的logits进行重加权，基于从下游数据中推断出的标签先验估计值进行去偏。
• 随后，通过学习到的融合权重，将去偏的零样本模型与微调模型进行集成，以提升鲁棒性和准确率。
• 该方法适用于多种微调范式：端到端微调、线性探测和提示微调。

Figure 1 : (a) Per class accuracy of CLIP-ViT/B16 on ImageNet. Class index are sorted using the estimated pre-training label prior. Curves are smoothed for better visualization. (b) Beak-down performance of different models on ImageNet. We equally divide the ImageNet classes into three subgroups, ac

实验结果

研究问题

• RQ1 由偏斜预训练数据引起的标签偏差是否会导致基础模型在下游任务中性能下降，尤其是在稀有类别上？
• RQ2 能否在不访问原始预训练数据的情况下，有效估计并纠正标签偏差？
• RQ3 将去偏的零样本模型与微调模型进行集成，是否能获得优于标准集成或仅微调的性能？
• RQ4 GLA在多种下游设置下的表现如何，包括少样本、多样本和长尾学习？
• RQ5 所提出的去偏方法在类别平衡评估下是否具有统计最优性？

主要发现

• GLA在ImageNet上实现1.5个百分点（pp）的准确率提升，显著改善了尾部类别的性能。
• 在11个少样本数据集上，GLA平均提升1.4至4.6个百分点，稀有类别上提升尤为显著。
• 在长尾分类任务中，GLA实现2.4个百分点的性能提升，展现出对类别不平衡的强大鲁棒性。
• 在DomainNet的草图域上，GLA将零样本准确率提升至93.00%，优于标准零样本（92.25%）和使用预训练标签先验的去偏方法（89.50%）。
• 该方法在三种微调范式——端到端、线性探测和提示微调——下均表现稳健，持续获得性能增益。
• GLA的性能优于WiSE-FT和ProGrad，尤其在尾部类别上，后两者方法存在性能下降问题。

Figure 2 : Illustration of debiasing process on ImageNet validation set. (a) The original distribution of zero-shot outputs; (b) the estimated pre-train distribution $\mathbf{q}$ based on our algorithm; (c) the distribution of debiased zero-shot outputs using estimated $\mathbf{q}$ .