[Paper Review] Model adaptation and unsupervised learning with non-stationary batch data under smooth concept drift

Subhro Das, Prasanth Lade+1 more|arXiv (Cornell University)|Feb 10, 2020

Data Stream Mining Techniques27 references4 citations

TL;DR

This paper proposes an unsupervised, iterative algorithm for adapting predictive models to non-stationary batch data under smooth concept drift, using a quantitative measure of posterior probability drift. The method improves classification accuracy over unadapted models and achieves comparable or better performance than state-of-the-art methods like SGARC, with significantly reduced runtime—demonstrated on synthetic and real-world manufacturing data.

ABSTRACT

Most predictive models assume that training and test data are generated from a stationary process. However, this assumption does not hold true in practice. In this paper, we consider the scenario of a gradual concept drift due to the underlying non-stationarity of the data source. While previous work has investigated this scenario under a supervised-learning and adaption conditions, few have addressed the common, real-world scenario when labels are only available during training. We propose a novel, iterative algorithm for unsupervised adaptation of predictive models. We show that the performance of our batch adapted prediction algorithm is better than that of its corresponding unadapted version. The proposed algorithm provides similar (or better, in most cases) performance within significantly less run time compared to other state of the art methods. We validate our claims though extensive numerical evaluations on both synthetic and real data.

Motivation & Objective

• To address the challenge of model adaptation in real-world scenarios where labels are unavailable during deployment, despite training data being labeled.
• To develop a quantitative definition of concept drift based on changes in posterior probability distributions across data points.
• To design an efficient, iterative algorithm that estimates point-wise drift and adapts the classifier without requiring labels during adaptation.
• To validate the method on both synthetic and real-world datasets, showing improved accuracy and reduced computational cost.
• To demonstrate that the proposed method outperforms or matches state-of-the-art approaches like SGARC in accuracy while being significantly faster.

Proposed method

• The method quantifies concept drift by measuring changes in the posterior probability distribution of each data point between initial (training) and drifted (test) data.
• It formulates an iterative optimization process that adjusts model parameters using only the test data and the previously trained model, without access to labels.
• The algorithm uses a step size schedule γ^(k) = 50 / √(k+1) and a convergence threshold of 10^−10 to ensure stable updates.
• It estimates the importance of each data point in drift estimation by analyzing shifts in class posterior probabilities.
• The adaptation process is applied to a Naive Bayes classifier, but the framework is generalizable to other classifiers.
• The method is evaluated using a batch adaptation setting, where only the model and test data are available post-training.

Experimental results

Research questions

• RQ1 How can concept drift be quantitatively measured in classification tasks under non-stationary data streams?
• RQ2 What is the impact of point-wise drift estimation on model performance when labels are unavailable during adaptation?
• RQ3 Can an unsupervised, iterative adaptation algorithm achieve better or comparable accuracy than supervised state-of-the-art methods with significantly less runtime?
• RQ4 How does the proposed method perform on real-world manufacturing data with gradual, smooth concept drift?
• RQ5 What is the trade-off between adaptation accuracy and computational efficiency in batch-mode model adaptation under drift?

Key findings

• On the UG_2C_2D dataset, the proposed algorithm achieved 96.08% classification accuracy, outperforming SGARC (1NN) at 95.56% and SGARC (SVM) at 95.53%.
• On the UG_2C_3D dataset, the method achieved 95.11% accuracy, surpassing SGARC (1NN) at 94.77% and SGARC (SVM) at 94.79%.
• On the UG_2C_5D dataset, the proposed method achieved 93.65% accuracy, exceeding SGARC (1NN) at 90.98% and SGARC (SVM) at 88.24%.
• The proposed algorithm ran in 5.20–5.50 seconds on average per dataset, compared to 10.11–120.21 seconds for SGARC (SVM), showing a significant speed advantage.
• On the real manufacturing dataset, the adapted model outperformed the unadapted model and was competitive with the supervised baseline, demonstrating effectiveness in real-world drift scenarios.
• The algorithm converged within 10 iterations in all experiments, indicating fast convergence and low computational overhead.