[论文解读] Style Normalization and Restitution for Domain Generalization and Adaptation

Xin Jin, Cuiling Lan+2 more|arXiv (Cornell University)|Jan 3, 2021

Domain Adaptation and Few-Shot Learning参考文献 84被引用 8

一句话总结

该论文提出了一种风格归一化与恢复（SNR）模块，通过实例归一化过滤掉域特定的风格差异，并从残差中恢复与任务相关的判别性特征，从而提升计算机视觉中的域泛化和无监督域自适应性能。该方法通过双重建损失解耦风格无关特征与任务相关特征，在分类、语义分割和目标检测基准上实现了最先进性能。

ABSTRACT

For many practical computer vision applications, the learned models usually have high performance on the datasets used for training but suffer from significant performance degradation when deployed in new environments, where there are usually style differences between the training images and the testing images. An effective domain generalizable model is expected to be able to learn feature representations that are both generalizable and discriminative. In this paper, we design a novel Style Normalization and Restitution module (SNR) to simultaneously ensure both high generalization and discrimination capability of the networks. In the SNR module, particularly, we filter out the style variations (e.g, illumination, color contrast) by performing Instance Normalization (IN) to obtain style normalized features, where the discrepancy among different samples and domains is reduced. However, such a process is task-ignorant and inevitably removes some task-relevant discriminative information, which could hurt the performance. To remedy this, we propose to distill task-relevant discriminative features from the residual (i.e, the difference between the original feature and the style normalized feature) and add them back to the network to ensure high discrimination. Moreover, for better disentanglement, we enforce a dual causality loss constraint in the restitution step to encourage the better separation of task-relevant and task-irrelevant features. We validate the effectiveness of our SNR on different computer vision tasks, including classification, semantic segmentation, and object detection. Experiments demonstrate that our SNR module is capable of improving the performance of networks for domain generalization (DG) and unsupervised domain adaptation (UDA) on many tasks. Code are available at https://github.com/microsoft/SNR.

研究动机与目标

• 解决由于训练与测试环境之间光照、色彩对比等风格差异导致的计算机视觉中的域偏移问题。
• 在不依赖目标域标签的设定下（如域泛化DG设置）提升模型泛化能力。
• 通过保留判别性特征并减少域特定风格差异，缓解无监督域自适应（UDA）中的性能下降问题。
• 通过新颖的特征重建机制，实现对任务相关与无关特征更优的解耦。
• 开发一种即插即用模块，在无需架构重构的前提下增强深度网络的泛化与判别能力。

提出的方法

• 对输入特征应用实例归一化（IN），以归一化风格差异（如光照、对比度），降低域间差异。
• 计算原始特征与归一化后特征之间的残差，以捕捉归一化过程中丢失的与任务相关的判别性信息。
• 通过将残差重新加回，重建原始特征表示，从而恢复判别能力。
• 引入双重建损失，通过最小化无关特征分量的熵和最大化相关特征分量的熵，促进特征解耦。
• 在训练过程中采用双分支结构计算重建损失，推理阶段丢弃辅助分支。
• 将SNR模块插入ResNet和DRN主干网络的卷积块之后，实现多种视觉任务的端到端训练。

Figure 1: Due to the differences in environments (such as lighting/camera/place/weather), the captured images present style discrepancy, such as the illumination, color contrast/saturation, quality, imaging style. These result in domain gaps between the training and testing data.

实验结果

研究问题

• RQ1 一个统一模块是否能在域偏移下同时提升深度网络的泛化与判别能力？
• RQ2 在风格归一化后，如何有效恢复任务相关特征以防止性能损失？
• RQ3 解耦风格无关与任务相关特征在多大程度上能提升域泛化与自适应性能？
• RQ4 所提出的双重建损失是否能带来比标准归一化或重建方法更优的特征解耦效果？
• RQ5 SNR模块是否能有效应用于分类、分割和检测等多种视觉任务？

主要发现

• 在mini-DomainNet基准上，SNR模块实现了最先进性能，域泛化设置下准确率最高提升2.1%。
• 在完整DomainNet数据集上，SNR增强的ResNet-152在域泛化设置下比基线模型高出1.8%的top-1准确率。
• 在语义分割任务中，使用DRN-D-105时，SNR模块在GTA5-to-Cityscapes设置下mIoU提升3.4%，在SYNTHIA-to-Cityscapes设置下提升2.9%。
• 在Cityscapes-to-Foggy Cityscapes和Cityscapes-to-KITTI基准上的目标检测任务中，SNR模块分别使mAP提升2.7%和2.1%。
• 消融实验表明，双重建损失显著提升了特征解耦效果，相比单损失变体，准确率提升1.5%。
• SNR模块在所有评估任务——分类、分割与检测中均一致提升性能，展现出广泛的适用性。

Figure 2: Overall flowchart. (a) Our generalizable feature learning network with the proposed Style Normalization and Restitution (SNR) module being plugged in after some convolutional blocks. Here, we use ResNet-50 as our backbone for illustration. (b) Proposed SNR module. Instance Normalization (I