[Paper Review] Continual Learning in the Frequency Domain

Ruiqi Liu, Boyu Diao+4 more|arXiv (Cornell University)|Oct 9, 2024

Seismology and Earthquake Studies5 citations

TL;DR

CLFD uses wavelet-based frequency-domain encoding and class-aware feature selection to improve rehearsal-based continual learning, boosting accuracy and drastically improving training efficiency on edge devices.

ABSTRACT

Continual learning (CL) is designed to learn new tasks while preserving existing knowledge. Replaying samples from earlier tasks has proven to be an effective method to mitigate the forgetting of previously acquired knowledge. However, the current research on the training efficiency of rehearsal-based methods is insufficient, which limits the practical application of CL systems in resource-limited scenarios. The human visual system (HVS) exhibits varying sensitivities to different frequency components, enabling the efficient elimination of visually redundant information. Inspired by HVS, we propose a novel framework called Continual Learning in the Frequency Domain (CLFD). To our knowledge, this is the first study to utilize frequency domain features to enhance the performance and efficiency of CL training on edge devices. For the input features of the feature extractor, CLFD employs wavelet transform to map the original input image into the frequency domain, thereby effectively reducing the size of input feature maps. Regarding the output features of the feature extractor, CLFD selectively utilizes output features for distinct classes for classification, thereby balancing the reusability and interference of output features based on the frequency domain similarity of the classes across various tasks. Optimizing only the input and output features of the feature extractor allows for seamless integration of CLFD with various rehearsal-based methods. Extensive experiments conducted in both cloud and edge environments demonstrate that CLFD consistently improves the performance of state-of-the-art (SOTA) methods in both precision and training efficiency. Specifically, CLFD can increase the accuracy of the SOTA CL method by up to 6.83% and reduce the training time by 2.6$ imes$.

Motivation & Objective

• Motivate continual learning (CL) in resource-constrained environments and address training efficiency alongside forgetting.
• Propose a frequency-domain framework to compress inputs and manage feature reuse across tasks.
• Enable seamless integration with existing rehearsal-based CL methods by optimizing only input/output features.
• Promote data augmentation and reduce memory footprint to store and replay more samples.

Proposed method

• Transform input images to the frequency domain using Discrete Wavelet Transform (DWT) in a Frequency Domain Feature Encoder (FFE).
• Compress input feature maps to reduce FLOPs and memory, enabling more replay samples within fixed buffers.
• Apply Class-aware Frequency Domain Feature Selection (CFFS) to balance feature reuse and interference across classes by frequency-domain similarity.
• Compute class-wise frequency-domain feature similarities using low-frequency components and select/top-masked features for classification.
• Use Frequency Dropout and Semantic Dropout to adapt feature selection across tasks without introducing cross-task learnable parameters.
• Integrate CLFD with existing rehearsal-based CL methods by only modifying input and output feature processing, leaving core rehearsal mechanisms intact.

Experimental results

Research questions

• RQ1 How can frequency-domain representations reduce input size and preserve information for continual learning?
• RQ2 Can class-aware selection of frequency-domain features reduce interference and forgetting across tasks?
• RQ3 What are the efficiency gains (time, memory, FLOPs) when pairing CLFD with standard rehearsal-based CL methods on edge devices?
• RQ4 Does CLFD maintain or improve accuracy relative to state-of-the-art rehearsal-based CL methods on standard benchmarks?

Key findings

• CLFD can improve accuracy of state-of-the-art rehearsal-based CL methods by up to 6.83%.
• On edge devices, CLFD achieves up to 2.6x training speed improvements and up to 3.0x peak memory reduction across benchmarks.
• Using DWT preserves both spatial and frequency information, enabling effective data augmentation and input compression.
• Class-aware frequency-domain feature selection reduces cross-task interference and aligns features for semantically similar classes.
• Integrating CLFD with multiple baselines (ER, DER++, ER-ACE, CLS-ER) yields consistent accuracy gains and memory reductions across datasets.