Skip to main content
QUICK REVIEW

[论文解读] Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization

Alexandre Ramé, Corentin Dancette|arXiv (Cornell University)|Sep 7, 2021
Domain Adaptation and Few-Shot LearningComputer Science被引用 25
一句话总结

Fishr 引入一个梯度方差正则化,用以在训练域之间对齐域级梯度统计,从而提升在 DomainBed 基准测试上的 out-of-distribution 泛化能力。

ABSTRACT

Learning robust models that generalize well under changes in the data distribution is critical for real-world applications. To this end, there has been a growing surge of interest to learn simultaneously from multiple training domains - while enforcing different types of invariance across those domains. Yet, all existing approaches fail to show systematic benefits under controlled evaluation protocols. In this paper, we introduce a new regularization - named Fishr - that enforces domain invariance in the space of the gradients of the loss: specifically, the domain-level variances of gradients are matched across training domains. Our approach is based on the close relations between the gradient covariance, the Fisher Information and the Hessian of the loss: in particular, we show that Fishr eventually aligns the domain-level loss landscapes locally around the final weights. Extensive experiments demonstrate the effectiveness of Fishr for out-of-distribution generalization. Notably, Fishr improves the state of the art on the DomainBed benchmark and performs consistently better than Empirical Risk Minimization. Our code is available at https://github.com/alexrame/fishr.

研究动机与目标

  • 在训练域之间的分布变化下,推动鲁棒的 OOD 泛化。
  • 提出一个梯度空间正则化器,使域级梯度方差匹配。
  • 证明对齐梯度方差可以对齐域级风险和 Hessian。
  • 展示在 DomainBed 和 Colored MNIST 上,相较于 ERM 和最先进方法的性能提升。

提出的方法

  • 定义相对于网络权重的损失的逐样本梯度 g_e^i。
  • 在每个域内的样本之间计算域级梯度方差 v_e = Var(G_e)。
  • 通过最小化域方差之间的距离进行正则化:L_Fishr = (1/|E|) sum_e ||v_e - v||^2,其中 v 是跨域的平均方差。
  • 可选地将 Fishr 仅应用于分类器权重 ω(或其他子集)以减少计算量。
  • 将梯度方差与 Hessian 对角线以及 Fisher 信息相关联,以证明匹配域级 Hessians 的合理性。
  • 使用 BackPACK 高效地计算用于正则化的逐样本梯度。
Figure 1 : Fishr principle . Fishr considers the individual (per-sample) gradients of the loss in the network weights $\theta$ . Specifically, Fishr matches the domain-level gradient variances of the distributions across the two training domains: $A$ ( $\{{\bm{g}}_{A}^{i}\}_{i=1}^{n_{A}}$ in orange)
Figure 1 : Fishr principle . Fishr considers the individual (per-sample) gradients of the loss in the network weights $\theta$ . Specifically, Fishr matches the domain-level gradient variances of the distributions across the two training domains: $A$ ( $\{{\bm{g}}_{A}^{i}\}_{i=1}^{n_{A}}$ in orange)

实验结果

研究问题

  • RQ1在训练域之间匹配域级梯度方差是否能够改善 out-of-distribution 泛化?
  • RQ2Fishr 是否也能对齐域级风险和 Hessians,减少域之间的损失景观不一致性?
  • RQ3梯度方差匹配是否比以往的梯度均值或基于特征的不变性方法更有效或更具可扩展性?
  • RQ4在像 Colored MNIST 和 DomainBed 这样的合成与真实域基准上,Fishr 相对于 ERM 和 IRM 的表现如何?

主要发现

  • 相对于 ERM 和若干基线,Fishr 提升了 DomainBed 基准上的 OOD 泛化。
  • 在收敛时,匹配域级梯度方差可以对齐域级风险和 Hessians。
  • 梯度方差作为 Hessian 对角线的代理,使正则化更高效。
  • 当应用于所有权重或仅应用于分类器权重时,Fishr 在 Colored MNIST 上达到最佳权衡。
  • 经验结果表明,在多个真实数据集(PACS、VLCS、OfficeHome、TerraIncognita、DomainNet)上,Fishr 始终优于 ERM。
Figure 2 : Loss landscapes around inconsistent weights $\theta^{*}$ at convergence. $N_{A,\theta^{*}}^{0.2}$ contains weights $\theta$ for which $\mathcal{R}_{A}(\theta)$ is low ( $\leq 0.2$ ) but $\mathcal{R}_{B}(\theta)$ is high ( $\geq 0.9$ ). This inconsistency is due to conflicting domain-level
Figure 2 : Loss landscapes around inconsistent weights $\theta^{*}$ at convergence. $N_{A,\theta^{*}}^{0.2}$ contains weights $\theta$ for which $\mathcal{R}_{A}(\theta)$ is low ( $\leq 0.2$ ) but $\mathcal{R}_{B}(\theta)$ is high ( $\geq 0.9$ ). This inconsistency is due to conflicting domain-level

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。