[Paper Review] SCLIFD:Supervised Contrastive Knowledge Distillation for Incremental Fault Diagnosis under Limited Fault Data

Peng Peng, Hanrong Zhang+4 more|arXiv (Cornell University)|Feb 12, 2023

Imbalanced Data Classification Techniques4 citations

TL;DR

This paper proposes SCLIFD, a novel framework for incremental fault diagnosis under limited fault data using supervised contrastive knowledge distillation, adaptive exemplar selection (AdaHerding), and a cosine classifier. It significantly outperforms existing methods, achieving 88.17% accuracy on imbalanced and 79.81% on long-tailed fault diagnosis datasets, demonstrating robustness to class imbalance and catastrophic forgetting in industrial settings.

ABSTRACT

Intelligent fault diagnosis has made extraordinary advancements currently. Nonetheless, few works tackle class-incremental learning for fault diagnosis under limited fault data, i.e., imbalanced and long-tailed fault diagnosis, which brings about various notable challenges. Initially, it is difficult to extract discriminative features from limited fault data. Moreover, a well-trained model must be retrained from scratch to classify the samples from new classes, thus causing a high computational burden and time consumption. Furthermore, the model may suffer from catastrophic forgetting when trained incrementally. Finally, the model decision is biased toward the new classes due to the class imbalance. The problems can consequently lead to performance degradation of fault diagnosis models. Accordingly, we introduce a supervised contrastive knowledge distillation for incremental fault diagnosis under limited fault data (SCLIFD) framework to address these issues, which extends the classical incremental classifier and representation learning (iCaRL) framework from three perspectives. Primarily, we adopt supervised contrastive knowledge distillation (KD) to enhance its representation learning capability under limited fault data. Moreover, we propose a novel prioritized exemplar selection method adaptive herding (AdaHerding) to restrict the increase of the computational burden, which is also combined with KD to alleviate catastrophic forgetting. Additionally, we adopt the cosine classifier to mitigate the adverse impact of class imbalance. We conduct extensive experiments on simulated and real-world industrial processes under different imbalance ratios. Experimental results show that our SCLIFD outperforms the existing methods by a large margin.

Motivation & Objective

• Address the challenge of class-incremental fault diagnosis under limited and imbalanced fault data in industrial systems.
• Overcome catastrophic forgetting during incremental model training when new fault classes emerge.
• Enhance representation learning and mitigate class imbalance effects in low-data regimes.
• Reduce computational cost during incremental learning by optimizing exemplar selection.
• Develop a scalable, robust framework for real-world industrial fault diagnosis with evolving fault classes.

Proposed method

• Introduces supervised contrastive knowledge distillation (SCL-KD) to improve feature representation learning under limited fault data.
• Proposes AdaHerding, a prioritized exemplar selection method that selects hard-to-learn samples based on learning difficulty to reduce memory and computational overhead.
• Integrates knowledge distillation with AdaHerding to alleviate catastrophic forgetting during incremental learning.
• Employs a cosine classifier at inference to reduce bias toward majority classes and improve generalization under class imbalance.
• Extends the iCaRL framework by incorporating SCL-KD, AdaHerding, and cosine classification for improved incremental fault diagnosis.
• Uses a multi-layer perceptron encoder with 24, 12, and 10 neurons for feature extraction in both simulated (TEP) and real-world (MFF) datasets.

Experimental results

Research questions

• RQ1 Can supervised contrastive knowledge distillation improve representation learning in low-data fault diagnosis scenarios?
• RQ2 How effectively can AdaHerding reduce computational cost while maintaining model performance in incremental fault diagnosis?
• RQ3 To what extent does the cosine classifier mitigate performance degradation due to class imbalance in fault diagnosis?
• RQ4 How does SCLIFD compare to state-of-the-art methods in terms of accuracy and stability across incremental sessions under imbalanced and long-tailed data distributions?
• RQ5 Can the proposed framework prevent catastrophic forgetting while maintaining high performance on previously learned fault classes?

Key findings

• SCLIFD achieves 88.17% test accuracy on the imbalanced fault diagnosis setting of the MFF dataset, outperforming the second-best method (iCaRL) by 33.15%.
• On the long-tailed fault diagnosis setting of the MFF dataset, SCLIFD reaches 79.81% accuracy, surpassing iCaRL by 35.84%.
• In ablation studies, each component—SCL, AdaHerding, and cosine classifier—contributes significantly, with individual gains of 21.86%, 6.42%, and 19.69% over iCaRL in session 5 under imbalanced conditions.
• SCLIFD consistently maintains the highest accuracy across all incremental sessions and remains closest to the upper bound performance, indicating strong stability and scalability.
• The framework demonstrates superior robustness in both simulated (TEP) and real-world (MFF) industrial datasets, confirming its practical viability.
• The integration of SCL-KD and AdaHerding effectively mitigates catastrophic forgetting, preserving performance on previously learned fault classes during incremental learning.