[论文解读] Efficient Test-Time Model Adaptation without Forgetting

Shuaicheng Niu, Jiaxiang Wu+5 more|arXiv (Cornell University)|Apr 6, 2022

Domain Adaptation and Few-Shot Learning被引用 66

一句话总结

提出 EATA，一种高效的全测试时自适应方法，具主动样本选择和 Fisher 正则化，在提高 OOD 性能的同时减轻对 ID 数据的遗忘。它仅更新 BN 的仿射参数，并使用基于熵的与基于多样性的准则来选择用于自适应的样本。

ABSTRACT

Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and testing data by adapting a given model w.r.t. any testing sample. This task is particularly important for deep models when the test environment changes frequently. Although some recent attempts have been made to handle this task, we still face two practical challenges: 1) existing methods have to perform backward computation for each test sample, resulting in unbearable prediction cost to many applications; 2) while existing TTA solutions can significantly improve the test performance on out-of-distribution data, they often suffer from severe performance degradation on in-distribution data after TTA (known as catastrophic forgetting). In this paper, we point out that not all the test samples contribute equally to model adaptation, and high-entropy ones may lead to noisy gradients that could disrupt the model. Motivated by this, we propose an active sample selection criterion to identify reliable and non-redundant samples, on which the model is updated to minimize the entropy loss for test-time adaptation. Furthermore, to alleviate the forgetting issue, we introduce a Fisher regularizer to constrain important model parameters from drastic changes, where the Fisher importance is estimated from test samples with generated pseudo labels. Extensive experiments on CIFAR-10-C, ImageNet-C, and ImageNet-R verify the effectiveness of our proposed method.

研究动机与目标

• Motivation: address distribution shifts between training and testing without access to training data or heavy backpropagation.
• Goal: achieve efficient test-time adaptation that maintains performance on in-distribution data.
• Aim: reduce backward computations per test sample while improving out-of-distribution generalization.
• Aim: prevent catastrophic forgetting during adaptation using a Fisher information-based regularizer.

提出的方法

• Introduce sample-adaptive entropy minimization with an active sample score S(x) to select reliable and non-redundant test samples for backward updates.
• Compute S(x) as a product of entropy-based weight S_ent(x) and diversity-based weight S_div(x) to filter out high-entropy and redundant samples.
• Update only the affine parameters of batch normalization layers during adaptation.
• Propose an anti-forgetting regularizer R( tildeTheta, tildeTheta^o ) using a Fisher information-based importance ω(θ_i) to constrain important parameters from large changes.
• Estimate Fisher importance ω(θ_i) from a small set of pseudo-labeled ID test samples constructed from the original model predictions.
• Combine entropy minimization with the regularizer in the overall objective for adaptation.
• Provide a lifelong/adaptation setting where parameters are not reset after each test batch.

实验结果

研究问题

• RQ1 How can test-time adaptation be made efficient by updating only a subset of parameters and selective samples?
• RQ2 Can entropy-based sample selection and diversity filtering improve robustness to OOD while reducing computation?
• RQ3 Does a Fisher-based regularizer prevent forgetting of ID performance during test-time adaptation?
• RQ4 How does the proposed method perform on CIFAR-10-C, ImageNet-C, and ImageNet-R compared to prior TTA methods?

主要发现

• EATA achieves higher OOD accuracy than several baselines on ImageNet-C and ImageNet-R while reducing backward passes.
• ET A (efficient) reduces average backward passes compared to TTT and MEMO by excluding unreliable and redundant samples.
• EATA (lifelong) maintains ID clean accuracy while preserving or improving OOD performance during continual adaptation.
• Ablations show that removing high-entropy and redundant samples improves efficiency and OOD performance.
• Fisher regularization effectively mitigates forgetting without sacrificing OOD gains.
• Experiments demonstrate that EATA outperforms ETA and comparable methods in both efficiency and robustness across benchmarks.