[论文解读] Guarding Barlow Twins Against Overfitting with Mixed Samples

Wele Gedara Chaminda Bandara, Celso M. de Melo+1 more|arXiv (Cornell University)|Dec 4, 2023

Domain Adaptation and Few-Shot Learning被引用 5

一句话总结

该论文提出Mixed Barlow Twins，一种正则化方法，通过在输入空间应用MixUp数据增强，在特征空间中诱导线性插值，从而减轻Barlow Twins自监督学习框架中的过拟合问题。该方法通过基于混合样本交叉相关对齐的附加正则化损失，减少表示过拟合并提升泛化能力，从而在CIFAR-10、CIFAR-100、TinyImageNet、STL-10和ImageNet上提升了下游性能。

ABSTRACT

Self-supervised Learning (SSL) aims to learn transferable feature representations for downstream applications without relying on labeled data. The Barlow Twins algorithm, renowned for its widespread adoption and straightforward implementation compared to its counterparts like contrastive learning methods, minimizes feature redundancy while maximizing invariance to common corruptions. Optimizing for the above objective forces the network to learn useful representations, while avoiding noisy or constant features, resulting in improved downstream task performance with limited adaptation. Despite Barlow Twins' proven effectiveness in pre-training, the underlying SSL objective can inadvertently cause feature overfitting due to the lack of strong interaction between the samples unlike the contrastive learning approaches. From our experiments, we observe that optimizing for the Barlow Twins objective doesn't necessarily guarantee sustained improvements in representation quality beyond a certain pre-training phase, and can potentially degrade downstream performance on some datasets. To address this challenge, we introduce Mixed Barlow Twins, which aims to improve sample interaction during Barlow Twins training via linearly interpolated samples. This results in an additional regularization term to the original Barlow Twins objective, assuming linear interpolation in the input space translates to linearly interpolated features in the feature space. Pre-training with this regularization effectively mitigates feature overfitting and further enhances the downstream performance on CIFAR-10, CIFAR-100, TinyImageNet, STL-10, and ImageNet datasets. The code and checkpoints are available at: https://github.com/wgcban/mix-bt.git

研究动机与目标

• 为解决Barlow Twins中特征过拟合的问题，尽管预训练性能强大，但其下游性能仍会下降。
• 探究在监督学习中常用的MixUp正则化是否可有效适配至非对比性、基于InfoMax的自监督学习方法（如Barlow Twins）。
• 通过在特征空间中引入基于线性插值的正则化项，增强样本间交互性与表示泛化能力。
• 通过实证验证所提方法在多种数据集和主干网络架构上的有效性。
• 提供对Barlow Twins的极简代码扩展，实现一致的性能提升，且超参数调优需求极少。

提出的方法

• 该方法引入一种基于MixUp的正则化损失，假设输入空间中的线性插值会引发特征空间中的线性插值。
• 对于每个批次，使用参数为α的Beta分布混合系数，随机混合两个样本，生成混合输入 y_m = α * y_a + (1 - α) * y_b。
• 网络为混合样本生成嵌入 z_m，并计算 z_m 与原始视图 z_a 和 z_b 的交叉相关矩阵。
• 基于特征空间中线性插值的假设，构建一个作为原始交叉相关矩阵加权平均的真值交叉相关矩阵。
• 正则化损失定义为预测交叉相关矩阵与真值交叉相关矩阵之间的L2距离，附加到原始Barlow Twins损失中。
• 该方法仅需对标准Barlow Twins实现进行少量代码修改，主要增加MixUp前向传播和附加损失项。

Figure 1 : Assessing the representation quality via $k$ -NN accuracy on the test-set during SSL training on train-set for information maximization-based Barlow Twins [ 74 ] vs. contrastive learning-based SimCLR [ 14 ] , on CIFAR-10 [ 42 ] dataset.

实验结果

研究问题

• RQ1 Barlow Twins目标在预训练过程中是否会出现过拟合，特别是在嵌入维度增加时？
• RQ2 通常用于监督学习的MixUp正则化，是否可有效适配至非对比性、基于InfoMax的自监督学习？
• RQ3 在输入空间中诱导线性插值是否能提升Barlow Twins中的特征泛化能力并减少过拟合？
• RQ4 在CIFAR-10、ImageNet和TinyImageNet等多样化数据集上，哪些超参数配置能实现最优性能？
• RQ5 所提出的Mixed Barlow Twins方法是否在多个基准测试中一致提升下游k-NN和线性探测准确率？

主要发现

• 在CIFAR-10和CIFAR-100上，当使用ResNet-18和投影器维度为1,024时，Mixed Barlow Twins相比标准Barlow Twins在k-NN准确率上实现了0.9%的绝对提升。
• 在TinyImageNet上，该方法相比基线将k-NN准确率提升了1.2%，最优性能出现在投影器维度为1,024且λ_reg = 4.0时。
• 在STL-10上，Mixed Barlow Twins实现了1.0%的k-NN准确率提升，λ_reg = 2.0时表现最佳。
• 在ImageNet上，该方法相比原始Barlow Twins将线性探测准确率提升了0.8%，最优配置为λ_reg = 1.0且投影器维度为8192。
• 该方法能一致防止在后期训练阶段的性能下降，而这种现象在标准Barlow Twins的小型数据集上已被观察到。
• 所有数据集上最优投影器维度均为1,024，λ_BT根据数据集设定为1/d或0.0078125。

Figure 2 : Schematic of the proposed Mixed Barlow Twins. (a) Original Barlow Twins Algorithm [ 74 ] . (b) Proposed MixUp regularization technique to prevent Barlow Twins from overfitting and to further enhance the representation quality.