[Paper Review] An Evaluation of Change Point Detection Algorithms

Gerrit J. J. van den Burg, Christopher K. I. Williams|arXiv (Cornell University)|Mar 13, 2020

Mental Health Research Topics87 citations

TL;DR

This paper creates a real-world, human-annotated benchmark dataset of 37 real time series (42 including quality-control series) for evaluating change point detection algorithms, defines evaluation metrics for multiple ground-truth annotations, and benchmarks 14 CPD methods under Default and Oracle settings.

ABSTRACT

Change point detection is an important part of time series analysis, as the presence of a change point indicates an abrupt and significant change in the data generating process. While many algorithms for change point detection have been proposed, comparatively little attention has been paid to evaluating their performance on real-world time series. Algorithms are typically evaluated on simulated data and a small number of commonly-used series with unreliable ground truth. Clearly this does not provide sufficient insight into the comparative performance of these algorithms. Therefore, instead of developing yet another change point detection method, we consider it vastly more important to properly evaluate existing algorithms on real-world data. To achieve this, we present a data set specifically designed for the evaluation of change point detection algorithms that consists of 37 time series from various application domains. Each series was annotated by five human annotators to provide ground truth on the presence and location of change points. We analyze the consistency of the human annotators, and describe evaluation metrics that can be used to measure algorithm performance in the presence of multiple ground truth annotations. Next, we present a benchmark study where 14 algorithms are evaluated on each of the time series in the data set. Our aim is that this data set will serve as a proving ground in the development of novel change point detection algorithms.

Motivation & Objective

• Provide a realistic benchmark dataset for evaluating change point detection algorithms on real-world time series.
• Develop evaluation metrics that account for multiple human ground-truth annotations.
• Benchmark a broad set of existing CPD methods to identify strengths, weaknesses, and failure modes on real data.

Proposed method

• Assemble a multi-domain real-world time series dataset (37 series, 42 including quality-control) with ground-truth change points annotated by five human labelers per series.
• Develop evaluation metrics that handle multiple ground-truth annotations, including a clustering-style covering metric and precision/recall with Margins of error (M=5).
• Provide an annotation tool and automated consistency checks to assess annotator agreement.
• Evaluate 14 CPD algorithms using two experimental settings: Default (using package defaults) and Oracle (grid-search maximum performance).
• Standardize series (zero mean, unit variance) to enable fair comparisons across methods.
• Report both univariate and multivariate results, and distinguish performance under realistic (Default) vs. optimistic (Oracle) parameter tuning.

Experimental results

Research questions

• RQ1 How well do existing CPD algorithms perform on a diverse, real-world benchmark with multiple human ground-truth annotations?
• RQ2 What evaluation metrics can robustly compare CPD methods when multiple annotators provide ground truth?
• RQ3 How do CPD methods compare under realistic defaults versus optimum (oracle) parameter settings across univariate and multivariate series?

Key findings

• A real-world CPD benchmark with 37 series (42 with quality-control series) and multi-annotator ground truth is feasible and informative.
• The study provides a practical evaluation framework with a clustering-based covering metric and a precision/recall F1 measure that accounts for multiple ground-truth annotations (M=5).
• Under Default settings, binary segmentation (binseg) and AMOC show strong performance on univariate data, while bocpdms leads on multivariate data (depending on metric).
• Under Oracle settings, bocpd achieves top performance on both univariate and multivariate data for the F1 score; rfpop shows strong performance on univariate data with covering metric.
• The baseline zero-change-point method often outperforms many methods under Default settings, highlighting the prevalence of false positives in practice.
• Annotator agreement is generally high (median around 0.8–0.9 depending on metric), with seasonal patterns posing the greatest challenge.