[论文解读] Contrastive Adapters for Foundation Model Group Robustness

Michael Zhang, Christopher Ré|arXiv (Cornell University)|Jul 14, 2022

Domain Adaptation and Few-Shot Learning被引用 12

一句话总结

本文提出对比适配器（contrastive adapters），一种在不微调基础模型（FM）的前提下提升其对群体分布偏移鲁棒性的方法。通过使用对比损失训练轻量级适配器，使同类别样本的嵌入在嵌入空间中更接近，不同类别样本的嵌入更远离，该方法在9个基准测试中将最差群体准确率提升了8.5至56.0个百分点，性能与完整模型微调相当，同时仅训练了≤1%的参数。

ABSTRACT

While large pretrained foundation models (FMs) have shown remarkable zero-shot classification robustness to dataset-level distribution shifts, their robustness to subpopulation or group shifts is relatively underexplored. We study this problem, and find that FMs such as CLIP may not be robust to various group shifts. Across 9 robustness benchmarks, zero-shot classification with their embeddings results in gaps of up to 80.7 percentage points (pp) between average and worst-group accuracy. Unfortunately, existing methods to improve robustness require retraining, which can be prohibitively expensive on large foundation models. We also find that efficient ways to improve model inference (e.g., via adapters, lightweight networks with FM embeddings as inputs) do not consistently improve and can sometimes hurt group robustness compared to zero-shot (e.g., increasing the accuracy gap by 50.1 pp on CelebA). We thus develop an adapter training strategy to effectively and efficiently improve FM group robustness. Our motivating observation is that while poor robustness results from groups in the same class being embedded far apart in the foundation model "embedding space," standard adapter training may not bring these points closer together. We thus propose contrastive adapting, which trains adapters with contrastive learning to bring sample embeddings close to both their ground-truth class embeddings and other sample embeddings in the same class. Across the 9 benchmarks, our approach consistently improves group robustness, raising worst-group accuracy by 8.5 to 56.0 pp over zero-shot. Our approach is also efficient, doing so without any FM finetuning and only a fixed set of frozen FM embeddings. On benchmarks such as Waterbirds and CelebA, this leads to worst-group accuracy comparable to state-of-the-art methods that retrain entire models, while only training $\leq$1% of the model parameters.

研究动机与目标

• 研究基础模型（FMs）在零样本分类任务中对子群体或群体分布偏移的鲁棒性。
• 解决在不进行昂贵的完整模型微调的前提下提升FM群体鲁棒性的挑战。
• 开发一种高效、基于适配器的方法，仅利用预训练FM的嵌入来提升鲁棒性。
• 克服标准适配器和线性探测方法中常见的不一致性和鲁棒性退化问题。
• 设计一种训练策略，显式地在嵌入空间中对齐同类别样本的嵌入。

提出的方法

• 在冻结的基础模型嵌入上训练轻量级适配器，以转换嵌入以实现更好的分类。
• 使用对比损失，促使同类别样本嵌入在嵌入空间中更接近。
• 应用监督对比损失，拉近正样本对（同类别）并推开负样本对（不同类别）。
• 在保持标准交叉熵损失以实现类别级别对齐的同时，在样本级别嵌入上添加对比正则化。
• 仅使用FM提供的嵌入端到端训练适配器，不微调基础模型。
• 通过仅训练适配器参数、冻结基础模型并固定嵌入，确保方法的高效性。

实验结果

研究问题

• RQ1 基础模型能否在群体分布偏移下保持对所有子群体的强零样本性能？
• RQ2 为何标准适配器和线性探测方法有时会降低群体鲁棒性而非提升？
• RQ3 对比学习能否有效应用于适配器训练，以改善嵌入空间中同类别样本的对齐？
• RQ4 是否可能在不微调基础模型的前提下显著提升最差群体准确率？
• RQ5 与需要完整模型微调的最先进方法相比，对比适配在鲁棒性和效率方面表现如何？

主要发现

• 在9个鲁棒性基准测试中，对比适配器相较于零样本分类，将最差群体准确率提升了8.5至56.0个百分点。
• 在CelebA数据集上，对比适配器实现了90.0%的最差群体准确率，优于零样本的74.0%和WiSE-FT的89.6%，超过所有基线方法。
• 在CelebA上，对比适配器将最差群体准确率差距缩小至仅0.7个百分点，而零样本为7.9个百分点。
• 在Waterbirds数据集上，该方法实现了83.7%的最差群体准确率，显著优于零样本（49.8%）和ERM适配器（60.8%）。
• 该方法在仅训练≤1%模型参数的情况下，性能与最先进的完整模型微调方法（如WiSE-FT）相当。
• 对比适配器在所有数据集上均一致提升了鲁棒性，即使在群体不平衡设置下，标准适配器失效的情况下也表现优异。