[Paper Review] A Fast and Efficient Change-point Detection Framework for Modern Data

Yiwei Liu, Hao Chen|arXiv (Cornell University)|Jun 24, 2020

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a fast, scalable change-point detection framework for modern high-dimensional and non-Euclidean data by leveraging approximate k-nearest neighbors to enable efficient computation with O(dn log n) time complexity. It introduces an analytic formula to control type I error and detects diverse change types in large-scale sequences, making it suitable for real-time and big data applications.

ABSTRACT

Change-point analysis is thriving in this big data era to address problems arising in many fields where massive data sequences are collected to study complicated phenomena over time. It plays an important role in processing these data by segmenting a long sequence into homogeneous parts for follow-up studies. The task requires the method to be able to process large datasets quickly and to deal with various types of changes for high-dimensional and non-Euclidean data. We propose a novel approach making use of approximate $k$-nearest neighbor information of the observations, and derive an analytic formula to control the type I error. The time complexity of the method is $O(dn\log n)$ for an $n$-length sequence of $d$-dimensional data. Moreover, we incorporate a useful pattern of data in high dimension that the proposed method could detect various types of changes in the sequence.

Motivation & Objective

• To address the challenge of detecting change-points in large-scale, high-dimensional, and non-Euclidean data sequences common in modern data science.
• To develop a method with low computational complexity suitable for real-time and streaming data applications.
• To enable detection of various types of changes—beyond simple mean shifts—across complex data structures.
• To control type I error rates analytically without relying on permutation or resampling methods.
• To maintain statistical power and accuracy while scaling efficiently with data size and dimensionality.

Proposed method

• The method uses approximate k-nearest neighbor (k-NN) graphs to capture local data structure and detect changes in sequence segments.
• It computes a test statistic based on the divergence of k-NN structure between adjacent windows in the data sequence.
• An analytic formula is derived to control the type I error rate under the null hypothesis of no change, eliminating the need for computationally expensive resampling.
• The approach leverages the intrinsic geometry of high-dimensional data, making it robust to non-Euclidean and complex data types.
• The algorithm achieves O(dn log n) time complexity by using efficient k-NN approximation techniques and incremental updates.
• It supports detection of multiple change types, including shifts in distribution, clustering structure, or manifold geometry.

Experimental results

Research questions

• RQ1 Can a change-point detection method achieve both high speed and high detection power on large-scale, high-dimensional data?
• RQ2 How can type I error be controlled analytically without permutation testing in complex data settings?
• RQ3 To what extent can approximate k-NN information capture diverse change types in non-Euclidean and high-dimensional sequences?
• RQ4 What is the computational complexity ceiling of a scalable, geometry-aware change-point detection framework?
• RQ5 How does the method perform across different types of data changes, such as distributional shifts or structural changes in manifolds?

Key findings

• The proposed method achieves O(dn log n) time complexity, making it suitable for large-scale data processing.
• The analytic type I error control formula ensures statistical validity without relying on computationally intensive resampling.
• The method effectively detects various change types, including non-location shifts, by exploiting local k-NN structure.
• The use of approximate k-NN enables scalability while preserving detection power in high-dimensional settings.
• The framework is applicable to non-Euclidean data due to its geometric foundation in neighborhood structure.
• Empirical results demonstrate strong performance in detecting subtle and complex changes across diverse data types.