[Paper Review] TKIL: Tangent Kernel Approach for Class Balanced Incremental Learning

Jinlin Xiang, Eli Shlizerman|arXiv (Cornell University)|Jun 17, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

TKIL proposes a novel class-balanced incremental learning method using a Gradients Tangent Kernel (GTK) loss to minimize divergence between feature representations across tasks, combined with an average weights update rule to prevent overfitting. It achieves state-of-the-art performance on CIFAR-100 and ImageNet, with near-perfect task prediction accuracy (up to 99.9%) and consistent class accuracy improvements over existing methods.

ABSTRACT

When learning new tasks in a sequential manner, deep neural networks tend to forget tasks that they previously learned, a phenomenon called catastrophic forgetting. Class incremental learning methods aim to address this problem by keeping a memory of a few exemplars from previously learned tasks, and distilling knowledge from them. However, existing methods struggle to balance the performance across classes since they typically overfit the model to the latest task. In our work, we propose to address these challenges with the introduction of a novel methodology of Tangent Kernel for Incremental Learning (TKIL) that achieves class-balanced performance. The approach preserves the representations across classes and balances the accuracy for each class, and as such achieves better overall accuracy and variance. TKIL approach is based on Neural Tangent Kernel (NTK), which describes the convergence behavior of neural networks as a kernel function in the limit of infinite width. In TKIL, the gradients between feature layers are treated as the distance between the representations of these layers and can be defined as Gradients Tangent Kernel loss (GTK loss) such that it is minimized along with averaging weights. This allows TKIL to automatically identify the task and to quickly adapt to it during inference. Experiments on CIFAR-100 and ImageNet datasets with various incremental learning settings show that these strategies allow TKIL to outperform existing state-of-the-art methods.

Motivation & Objective

• Address catastrophic forgetting in class incremental learning by maintaining balanced performance across all classes.
• Reduce bias toward recently learned tasks by preserving consistent feature representations across tasks.
• Enable accurate task prediction during inference to support effective fine-tuning to task-specific models.
• Develop a training strategy that prevents overfitting to the latest task despite class imbalance in memory.
• Introduce a novel loss function based on Neural Tangent Kernel (NTK) dynamics to coordinate feature representations across incremental learning steps.

Proposed method

• Propose a Gradients Tangent Kernel (GTK) loss that measures the distance between gradients of feature layers from previous and current tasks.
• Minimize the GTK loss during training to align feature representations across tasks and reduce divergence.
• Use an average weights update rule: train each task-specific model in a single mini-batch and average the weights to stabilize learning and prevent overfitting.
• Leverage Neural Tangent Kernel (NTK) theory to model gradient flow dynamics, even in finite-width networks, to guide representation alignment.
• Train a base model that can predict the task of incoming samples with high accuracy, enabling effective task-specific fine-tuning during inference.
• Integrate the GTK loss with knowledge distillation (KD) and memory replay to preserve knowledge from previous tasks.

Experimental results

Research questions

• RQ1 Can a kernel-based loss derived from gradient dynamics improve class-balanced performance in incremental learning?
• RQ2 How does the GTK loss affect the consistency and alignment of feature representations across sequentially learned tasks?
• RQ3 Does the average weights update rule mitigate overfitting to the latest task and improve generalization across classes?
• RQ4 To what extent can accurate task prediction during inference be achieved using the proposed method, and how does it impact fine-tuning performance?
• RQ5 How does TKIL compare to state-of-the-art methods in terms of overall accuracy, class-balanced accuracy, and task prediction reliability across multiple benchmarks?

Key findings

• TKIL achieves nearly 100% task prediction accuracy on CIFAR-100 across 10, 20, and 50 incremental stages, significantly outperforming baselines.
• On CIFAR-100 with 10 incremental stages, TKIL achieves 82.5% average class accuracy, surpassing the next best method by over 5 percentage points.
• With 20 incremental stages on CIFAR-100, TKIL maintains a stable accuracy curve with minimal drop (slope ~0.8), while other methods show significant performance collapse.
• The ablation study confirms that combining GTK loss with the average weights rule yields the highest class accuracy (82.5%) and task prediction accuracy (99.9%) on CIFAR-100.
• The optimal hyperparameter γ for GTK loss is γ = 0.1, as it balances representation retention and prevents overfitting to early or late tasks.
• On ImageNet, TKIL demonstrates strong generalization, outperforming existing SOTA methods in multi-class incremental learning scenarios.