[Paper Review] EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization

Junha Song, Jungsoo Lee+2 more|arXiv (Cornell University)|Mar 3, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

EcoTTA proposes a memory-efficient continual test-time adaptation method that uses lightweight meta networks and self-distilled regularization to reduce activation memory by 80–86% compared to SOTA methods like CoTTA, while preventing catastrophic forgetting and error accumulation through knowledge distillation from frozen original networks.

ABSTRACT

This paper presents a simple yet effective approach that improves continual test-time adaptation (TTA) in a memory-efficient manner. TTA may primarily be conducted on edge devices with limited memory, so reducing memory is crucial but has been overlooked in previous TTA studies. In addition, long-term adaptation often leads to catastrophic forgetting and error accumulation, which hinders applying TTA in real-world deployments. Our approach consists of two components to address these issues. First, we present lightweight meta networks that can adapt the frozen original networks to the target domain. This novel architecture minimizes memory consumption by decreasing the size of intermediate activations required for backpropagation. Second, our novel self-distilled regularization controls the output of the meta networks not to deviate significantly from the output of the frozen original networks, thereby preserving well-trained knowledge from the source domain. Without additional memory, this regularization prevents error accumulation and catastrophic forgetting, resulting in stable performance even in long-term test-time adaptation. We demonstrate that our simple yet effective strategy outperforms other state-of-the-art methods on various benchmarks for image classification and semantic segmentation tasks. Notably, our proposed method with ResNet-50 and WideResNet-40 takes 86% and 80% less memory than the recent state-of-the-art method, CoTTA.

Motivation & Objective

• Address the critical challenge of high memory consumption in test-time adaptation (TTA) on edge devices, where activation memory dominates during backpropagation.
• Overcome catastrophic forgetting and error accumulation in long-term continual TTA, where models degrade on source domains and accumulate errors due to unsupervised loss optimization.
• Design a memory-efficient architecture that freezes the original network and adapts only lightweight meta networks to reduce activation storage.
• Introduce a self-distilled regularization technique that preserves source domain knowledge by constraining meta network outputs to stay close to the frozen original network’s outputs.
• Achieve state-of-the-art performance with minimal memory overhead, enabling practical deployment in real-world, continuously changing environments.

Proposed method

• Propose a two-part architecture: a frozen pre-trained backbone network and lightweight meta networks (one BN layer and one convolution block) attached to specific stages.
• Update only the meta networks during test-time adaptation, avoiding backpropagation through the main network and drastically reducing activation memory storage.
• Implement self-distilled regularization that minimizes the L2 distance between the output of the meta network and the frozen original network’s output at the same layer.
• Apply the regularization in parallel with standard unsupervised losses (e.g., entropy minimization), requiring no additional memory or computation.
• Use a model partition factor K to control the number of layers adapted, enabling trade-offs between performance and memory usage.
• Train meta networks using a combined loss: unsupervised adaptation loss (e.g., entropy minimization) and the self-distillation regularization loss.

Figure 1 : (a) Memory cost comparison between TTA methods . The size of activations, not the parameters, is the primary memory bottleneck during training. (b) CIFAR-C adaptation performance. We perform the continual online adaptation on CIFAR-C dataset. The x- and y-axis are the average error of all

Experimental results

Research questions

• RQ1 Can a lightweight meta network architecture significantly reduce memory consumption in continual TTA while maintaining performance?
• RQ2 Does self-distilled regularization effectively prevent catastrophic forgetting and error accumulation in long-term continual adaptation?
• RQ3 How does EcoTTA compare to SOTA methods like CoTTA in terms of memory efficiency and accuracy on continual TTA benchmarks?
• RQ4 Can the proposed method achieve state-of-the-art performance with minimal memory overhead in real-world edge deployment scenarios?
• RQ5 What is the trade-off between adaptation performance and memory consumption when varying the number of adapted layers (via partition factor K)?

Key findings

• EcoTTA reduces activation memory consumption by 86% and 80% compared to CoTTA for ResNet-50 and WideResNet-40, respectively, under the same continual TTA setup.
• On CIFAR-100-C with severity 5, EcoTTA (K=4) achieves an average error of 36.4% with only 80 MB memory, outperforming CoTTA (38.1% error, 409 MB) and other SOTA methods in accuracy while using significantly less memory.
• The self-distilled regularization enables stable long-term adaptation, preventing performance degradation on source domains and reducing error accumulation over time.
• Even with only one BN and one convolution block per meta network, EcoTTA achieves competitive performance, demonstrating that minimal parameter updates can yield substantial gains.
• The method incurs negligible computational overhead from regularization, as it is computed in parallel with the adaptation loss and requires no extra memory.
• EcoTTA maintains strong generalization across diverse corruptions in the CIFAR-C benchmark, showing robustness in continual adaptation under domain shift.

Figure 2 : Architecture for test-time adaptation. We illustrate TTA methods: TENT [ 65 ] , EATA [ 50 ] , CoTTA [ 66 ] , and Ours (EcoTTA). TENT and EATA update multiple batch norm layers, in which large activations have to be stored for gradient calculation. In CoTTA, an entire network is trained wi