[论文解读] One Pool Is Not Enough: Multi-Cluster Memory for Practical Test-Time Adaptation
提出多簇记忆（MCM），将记忆组织为基于描述符的多个簇以用于 Practical TTA，在与现有基于记忆的方法结合时，在多个数据集上获得一致的增益。
Test-time adaptation (TTA) adapts pre-trained models to distribution shifts at inference using only unlabeled test data. Under the Practical TTA (PTTA) setting, where test streams are temporally correlated and non-i.i.d., memory has become an indispensable component for stable adaptation, yet existing methods universally store amples in a single unstructured pool. We show that this single-cluster design is fundamentally mismatched to PTTA: a stream clusterability analysis reveals that test streams are inherently multi-modal, with the optimal number of mixture components consistently far exceeding one. To close this structural gap, we propose Multi-Cluster Memory (MCM), a plug-and-play framework that organizes stored samples into multiple clusters using lightweight pixel-level statistical descriptors. MCM introduces three complementary mechanisms: descriptor-based cluster assignment to capture distinct distributional modes, Adjacent Cluster Consolidation (ACC) to bound memory usage by merging the most similar temporally adjacent clusters, and Uniform Cluster Retrieval (UCR) to ensure balanced supervision across all modes during adaptation. Integrated with three contemporary TTA methods on CIFAR-10-C, CIFAR-100-C, ImageNet-C, and DomainNet, MCM achieves consistent improvements across all 12 configurations, with gains up to 5.00% on ImageNet-C and 12.13% on DomainNet. Notably, these gains scale with distributional complexity: larger label spaces with greater multi-modality benefit most from multi-cluster organization. GMM-based memory diagnostics further confirm that MCM maintains near-optimal distributional balance, entropy, and mode coverage, whereas single-cluster memory exhibits persistent imbalance and progressive mode loss. These results establish memory organization as a key design axis for practical test-time adaptation.
研究动机与目标
- 证明 PTTA 数据流本质上是多模态的，单簇记忆不足以覆盖其多样性。
- 引入带描述符聚类的 Multi-Cluster Memory (MCM)，包含 ACC 与 UCR。
- 证明在 PTTA 下，MCM 能在多种基线与数据集上提升自适应准确性。
- 提供诊断工具以量化记忆质量与模态覆盖。
- 强调记忆组织作为 Practical TTA 的关键设计维度。
提出的方法
- 将记忆描述为具有每簇容量的多簇结构。
- 使用像素级通道统计作为聚类的描述符。
- 引入 Adjacent Cluster Consolidation (ACC) 在记忆满时合并相邻的相似簇。
- 引入 Uniform Cluster Retrieval (UCR) 在自适应过程中从所有簇中均等抽样。
- 将 MCM 与 RoTTA、PeTTA、ResiTTA 集成，在 CIFAR-10-C、CIFAR-100-C、ImageNet-C 与 DomainNet 上评估。
- 提供基于 GMM 的诊断框架以评估记忆平衡、熵与模态覆盖。
实验结果
研究问题
- RQ1PTTA 数据流是否本质上是多模态的，单一记忆簇能否捕捉其多样性？
- RQ2结构化的多簇记忆是否能在 PTTA 下提升记忆型自适应的稳定性与性能？
- RQ3描述符聚类、ACC 与 UCR 是否共同带来比 SCM 更好的记忆覆盖与自适应效果？
主要发现
| 方法 | 场景 | CIFAR10-C | CIFAR100-C | ImageNet-C | DomainNet |
|---|---|---|---|---|---|
| Source | – | 43.50 | 46.40 | 82.00 | – |
| BN | CoRR’20 | 75.20 | 52.90 | – | – |
| PL | ICML’13 | 82.90 | 88.90 | – | – |
| TENT | ICLR’21 | 86.00 | 92.80 | – | – |
| LAME | CVPR’22 | 39.50 | 40.50 | 80.90 | – |
| CoTTA | CVPR’22 | 83.20 | 52.20 | 98.60 | – |
| NOTE | NeurIPS’22 | 31.10 | 73.80 | – | – |
| RDumb | NeurIPS’23 | 31.10 | 36.70 | 72.20 | 44.30 |
| ROID | WACV’24 | 72.70 | 76.40 | 62.70 | – |
| TRIBE | AAAI’24 | 15.30 | 33.80 | 63.60 | – |
| NEO | ICLR’26 | 46.36 | 43.25 | 78.25 | – |
| RoTTA | CVPR’23 | 25.20 | 35.00 | 68.30 | 44.30 |
| RoTTA + MCM | – | 22.59 | 33.75 | 67.46 | 42.53 |
| PeTTA | NeurIPS’24 | 24.30 | 35.80 | 65.30 | 43.80 |
| PeTTA + MCM | – | 21.55 | 33.04 | 60.30 | 42.80 |
| ResiTTA | ICASSP’25 | 22.80 | 32.50 | 69.40 | 54.76 |
| ResiTTA + MCM | – | 20.69 | 31.90 | 66.65 | 42.63 |
- MCM 在所有 12 个基线–数据集配置中均有提升，错误率平均下降 2.96%。
- 最大增益出现在与基线组合时：ImageNet-C 提升 5.00%，DomainNet 提升 12.13%。
- 描述符基于像素统计的聚类在此场景优于基于 CNN 的描述符。
- ACC 在准确性与运行时之间提供最佳折衷，优于 GCC、SCM 与 LRU 策略。
- 记忆诊断显示 MCM 实现了平衡的占用、高熵与完整的模态覆盖，而 SCM 则存在不平衡和模态丢失。
- 基于 GMM 的诊断 corroborate 多簇记忆在整个 PTTA 过程保留多模态分布结构。
更好的研究，从现在开始
从论文设计到论文写作，大幅缩短您的研究时间。
无需绑定信用卡
本解读由 AI 生成，并经人工编辑审核。