[Paper Review] A Memory Transformer Network for Incremental Learning

Ahmet İşcen, Thomas Bird+3 more|arXiv (Cornell University)|Oct 10, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes the Memory Transformer Network (MTN), a novel class-incremental learning method that enhances exemplar-based rehearsal by using a lightweight transformer to dynamically aggregate features from the k-nearest neighbors in a memory bank during inference. MTN achieves state-of-the-art accuracy on ImageNet-1k and Google-Landmarks-1k benchmarks by explicitly modeling query-memory relationships, outperforming prior methods even with small memory footprints.

ABSTRACT

We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from. Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes. One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks. In our paper, we propose to enhance the utilization of this memory bank: we not only use it as a source of additional training data like existing works but also integrate it in the prediction process explicitly.Our method, the Memory Transformer Network (MTN), learns how to combine and aggregate the information from the nearest neighbors in the memory with a transformer to make more accurate predictions. We conduct extensive experiments and ablations to evaluate our approach. We show that MTN achieves state-of-the-art performance on the challenging ImageNet-1k and Google-Landmarks-1k incremental learning benchmarks.

Motivation & Objective

• To address catastrophic forgetting in class-incremental learning, where models forget previously learned classes when trained on new data.
• To improve upon existing exemplar-based rehearsal methods that use memory only during training, not during inference.
• To enhance prediction accuracy by explicitly leveraging memory exemplars during decision-making, not just for training.
• To design a lightweight, efficient architecture that can scale to small memory budgets while maintaining high performance.
• To demonstrate that modeling local feature neighborhoods via a transformer leads to more robust and accurate predictions in incremental settings.

Proposed method

• MTN uses a k-nearest neighbors (kNN) search in feature space to retrieve the most relevant exemplars from a memory bank for a given query image.
• The query vector and its k nearest neighbors are processed by a lightweight transformer that models their interactions to produce a refined, context-aware representation.
• The transformer attends to the query and its neighbors, learning to re-weight and aggregate features to improve prediction confidence and accuracy.
• The final prediction is made based on the aggregated representation, with the output distribution conditioned on both the query and its memory neighbors.
• The method is applied at inference time, making it orthogonal to model training and compatible with any backbone and feature extractor.
• The architecture is efficient, as it only processes a small subset of the memory (k neighbors), avoiding the computational cost of attending to the full memory bank.

Experimental results

Research questions

• RQ1 Can explicitly incorporating memory exemplars into the prediction process improve generalization and reduce forgetting in class-incremental learning?
• RQ2 How does modeling the local neighborhood of a query in feature space via a transformer compare to standard kNN or classifier-based inference?
• RQ3 Does the proposed method maintain performance under small memory budgets, where exemplar efficiency is critical?
• RQ4 Can the MTN architecture outperform state-of-the-art methods on challenging benchmarks like ImageNet-1k and Google-Landmarks-1k?
• RQ5 How sensitive is MTN to hyperparameters such as k (number of neighbors) and memory size M?

Key findings

• MTN achieves state-of-the-art accuracy on the ImageNet-1k benchmark, outperforming prior methods including SS-IL and iCaRL.
• On Google-Landmarks-1k, MTN achieves the highest reported accuracy, demonstrating strong generalization on a fine-grained, open-vocabulary dataset.
• MTN is highly robust to small memory sizes, maintaining high performance even when M is as low as 10k, while baseline methods like SS-IL degrade significantly.
• The method shows stable performance across different k values (k ≥ 10), with k=10 selected as optimal for efficiency and accuracy.
• Qualitative analysis reveals that MTN successfully reweights misclassified neighbors, giving higher attention to correct-class exemplars even in ambiguous cases.
• The medium-sized transformer (4 layers, 128-dim) achieves optimal trade-off between performance and efficiency, with no benefit from larger architectures.