[Paper Review] Saliency Guided Experience Packing for Replay in Continual Learning

Gobinda Saha, Kaushik Roy|arXiv (Cornell University)|Sep 10, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Experience Packing and Replay (EPR), a continual learning method that selects saliency-guided image patches—key regions for model predictions—instead of full images for episodic memory. By zero-padding and replaying these informative patches, EPR improves memory efficiency and reduces catastrophic forgetting, achieving state-of-the-art accuracy with minimal memory size on CIFAR-100, miniImageNet, and CUB datasets.

ABSTRACT

Artificial learning systems aspire to mimic human intelligence by continually learning from a stream of tasks without forgetting past knowledge. One way to enable such learning is to store past experiences in the form of input examples in episodic memory and replay them when learning new tasks. However, performance of such method suffers as the size of the memory becomes smaller. In this paper, we propose a new approach for experience replay, where we select the past experiences by looking at the saliency maps which provide visual explanations for the model's decision. Guided by these saliency maps, we pack the memory with only the parts or patches of the input images important for the model's prediction. While learning a new task, we replay these memory patches with appropriate zero-padding to remind the model about its past decisions. We evaluate our algorithm on CIFAR-100, miniImageNet and CUB datasets and report better performance than the state-of-the-art approaches. With qualitative and quantitative analyses we show that our method captures richer summaries of past experiences without any memory increase, and hence performs well with small episodic memory.

Motivation & Objective

• Address catastrophic forgetting in continual learning by improving the informativeness of episodic memory without increasing its size.
• Overcome the performance degradation of experience replay in small-memory settings due to overfitting on limited samples.
• Leverage explainable AI (XAI) techniques, specifically saliency maps, to identify and store only the most relevant image regions for past task predictions.
• Design a memory packing mechanism that increases sample diversity per class while maintaining constant memory capacity.
• Demonstrate that replaying saliency-selected patches with zero-padding leads to better generalization and reduced forgetting compared to full-image replay.

Proposed method

• Use saliency maps (e.g., Grad-CAM) to identify localized, prediction-critical regions in input images after each task.
• Extract and store only the saliency-highlighted patches from each image, along with their spatial coordinates, in the episodic memory buffer.
• Introduce the Experience Packing Factor (EPF) to control the number of patches per class and the size of each patch.
• Construct composite memory images per class by combining multiple saliency patches, enhancing diversity without increasing memory footprint.
• During new task training, retrieve stored patches, zero-pad them to original image dimensions, and jointly train with current task data.
• Apply standard backpropagation and optimization on the combined loss from current task data and zero-padded memory patches.

Experimental results

Research questions

• RQ1 Can saliency-guided patch selection improve the representativeness of episodic memory in continual learning without increasing memory size?
• RQ2 How does the performance of EPR compare to state-of-the-art experience replay methods under varying memory constraints?
• RQ3 Does the choice of saliency method significantly affect the performance of the proposed EPR framework?
• RQ4 To what extent does EPR reduce overfitting and forgetting compared to full-image replay baselines?
• RQ5 What is the computational overhead of EPR, and does it remain efficient despite the added saliency and zero-padding operations?

Key findings

• EPR achieves state-of-the-art accuracy on CIFAR-100, miniImageNet, and CUB datasets, outperforming DER++, ER-RING, and MEGA-I across all memory sizes.
• On CIFAR-100, EPR achieves 58.5% accuracy with EPF=2, and performance drops when EPF increases beyond this value, indicating optimal patch size and number per class.
• On miniImageNet, the best performance is achieved at EPF=3, confirming that patch size and quantity are dataset-dependent and linked to saliency map quality.
• Models trained on EPR memory buffers generalize better than those on DER++ or ER-RING buffers, with the highest multitask accuracy in ablation studies.
• The method is robust to saliency method choice: performance varies by less than 1% across Grad-CAM, Grad-CAM++, Smooth-Grad, and FullGrad.
• EPR incurs only ~30% extra training time compared to standard ER-RING, while delivering ~7% accuracy gain, making it both efficient and effective.