[Paper Review] Testing for concept shift online

Vladimir Vovk|arXiv (Cornell University)|Dec 28, 2020

Data Stream Mining Techniques11 references4 citations

TL;DR

This paper proposes a novel method for online detection of concept shift in machine learning using decomposable exchangeability martingales derived from conformal prediction. By separating evidence for concept shift from label shift, the approach achieves significantly stronger detection performance—demonstrated by a 10^43 final martingale value on the USPS dataset—surpassing prior methods.

ABSTRACT

This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction.

Motivation & Objective

• To develop online methods for detecting concept shift in classification tasks where data distributions change over time.
• To decompose exchangeability martingales into components that isolate evidence for concept shift versus label shift.
• To improve upon existing exchangeability martingales by constructing martingales that are both highly sensitive to concept shift and perfectly decomposable.
• To provide a practical framework for deciding when to retrain machine learning models based on detectable distributional shifts.
• To extend conformal prediction techniques to detect dangerous shifts that invalidate ROC analysis in Y→X domains.

Proposed method

• Constructs exchangeability martingales that are martingales under any exchangeable distribution, enabling online hypothesis testing for violations of the IID assumption.
• Uses label-conditional conformity measures to generate p-values that are invariant to label permutations, ensuring exchangeability.
• Applies the Sleepy Jumper betting strategy to conformal p-values to generate nonnegative martingales that grow when distributional shifts occur.
• Decomposes the overall martingale into a product of two components: one detecting concept shift (via conditional p-values) and one detecting label shift (via marginal p-values).
• Employs the same-class conformity measure and nearest-object conformity measure to explore sensitivity of detection performance to conformity scoring design.
• Combines separate martingales for concept and label shift using Theorem 2, which guarantees the product remains a valid exchangeability martingale under the IID assumption.

Experimental results

Research questions

• RQ1 Can exchangeability martingales be constructed to detect concept shift specifically, rather than conflating it with label shift?
• RQ2 Is it possible to decompose an exchangeability martingale into components that separately detect concept shift and label shift?
• RQ3 How does the performance of a decomposable martingale compare to non-decomposable ones in detecting dataset shift?
• RQ4 Can improved detection of concept shift be achieved without sacrificing sensitivity to label shift?
• RQ5 What is the impact of different conformity measures on the final value of the exchangeability martingale in real-world datasets?

Key findings

• The product of a concept-shift-detecting martingale and a label-shift-detecting martingale forms a valid, perfectly decomposable exchangeability martingale.
• On the USPS dataset, the decomposable martingale achieved a final value of approximately 5.16 × 10^43, vastly exceeding the previous record of the black martingale.
• The red (concept shift) and green (label shift) martingales in Figure 1 show that the decomposable approach dominates the original black martingale after around 3000 observations.
• Using the same-class conformity measure significantly enhances detection of concept shift, though at the cost of reduced label shift detection performance.
• Replacing the denominator in the conformity score with the distance to the nearest neighbor (regardless of class) leads to a slight deterioration in concept shift detection, indicating that class-specific distance is more informative.
• Combining the green martingale from Figure 1 with the red martingale from Figure 2 yields a final martingale value of 5.16 × 10^43, demonstrating the power of component-wise optimization.