[Paper Review] Detecting Abrupt Changes in the Presence of Local Fluctuations and Autocorrelated Noise

Gaetano Romano, Guillem Rigaill+2 more|arXiv (Cornell University)|May 4, 2020

Gene expression and cancer classification57 references34 citations

TL;DR

This paper proposes DeCAFS, a novel dynamic programming algorithm that detects abrupt changes in univariate time series by modeling local mean fluctuations as a random walk and autocorrelated noise via an AR(1) process. By minimizing a penalized likelihood cost function with a BIC penalty, DeCAFS outperforms existing methods in detecting true changepoints under realistic conditions of autocorrelation and local variation, as demonstrated on well-log and bacterial gene expression data.

ABSTRACT

Whilst there are a plethora of algorithms for detecting changes in mean in univariate time-series, almost all struggle in real applications where there is autocorrelated noise or where the mean fluctuates locally between the abrupt changes that one wishes to detect. In these cases, default implementations, which are often based on assumptions of a constant mean between changes and independent noise, can lead to substantial over-estimation of the number of changes. We propose a principled approach to detect such abrupt changes that models local fluctuations as a random walk process and autocorrelated noise via an AR(1) process. We then estimate the number and location of changepoints by minimising a penalised cost based on this model. We develop a novel and efficient dynamic programming algorithm, DeCAFS, that can solve this minimisation problem; despite the additional challenge of dependence across segments, due to the autocorrelated noise, which makes existing algorithms inapplicable. Theory and empirical results show that our approach has greater power at detecting abrupt changes than existing approaches. We apply our method to measuring gene expression levels in bacteria.

Motivation & Objective

• To address the poor performance of standard changepoint detection methods when data exhibit autocorrelated noise and local mean fluctuations.
• To develop a principled statistical model that jointly accounts for both autocorrelation and local mean drift.
• To design an efficient dynamic programming algorithm capable of exactly minimizing the penalized cost under this complex model.
• To evaluate the method's performance on real-world data, particularly in gene expression analysis where both abrupt shifts and gradual drifts are biologically relevant.
• To demonstrate that DeCAFS provides more accurate and robust changepoint detection than existing approaches under model misspecification.

Proposed method

• Models local mean fluctuations as a random walk process and measurement noise as an AR(1) process.
• Defines a penalized cost function based on the log-likelihood of the full model plus a BIC penalty for each changepoint.
• Develops DeCAFS, a novel dynamic programming algorithm that extends functional pruning to handle dependence across segments due to AR(1) noise.
• Uses a pre-processing step to estimate model parameters, including variance and autocorrelation from k-lag differences in the data.
• Applies the functional pruning approach to efficiently compute the optimal segmentation under the full model.
• Employs a penalty inflation strategy based on theoretical results to maintain consistency under model error.

Experimental results

Research questions

• RQ1 Can a model that jointly accounts for local mean fluctuations and autocorrelated noise improve changepoint detection in real-world time series?
• RQ2 How does DeCAFS compare to existing methods like wild binary segmentation and AR1Seg in terms of accuracy and robustness?
• RQ3 Does the use of a full likelihood-based penalized cost with BIC penalty lead to better detection power than approximations or heuristic penalties?
• RQ4 Can DeCAFS maintain consistent estimation of changepoint number and location when the model is even slightly misspecified?
• RQ5 How does DeCAFS perform on real biological data, such as bacterial gene expression, where both abrupt shifts and gradual drifts are biologically meaningful?

Key findings

• DeCAFS significantly reduces overfitting compared to standard methods: on the well-log data, it detects 10 changepoints with a default penalty, while wild binary segmentation with standard criteria detects 25, and the AR(1) penalty-inflated method detects 18.
• DeCAFS achieves better alignment with biological ground truth than hmmTiling in detecting terminators, with M(δ) values for terminators consistently higher across all thresholds δ.
• For promoters, DeCAFS performs comparably to hmmTiling at low thresholds but outperforms it at higher thresholds, indicating greater robustness to noise.
• The method detects 10 changepoints in the well-log data with a default penalty, which aligns well with expert interpretation of rock strata transitions.
• On bacterial gene expression data, DeCAFS analyzes 192,000 data points in approximately 7 minutes, outperforming hmmTiling’s 5h36m runtime for both strands.
• Theoretical analysis confirms that DeCAFS maintains consistency in changepoint estimation even under model misspecification, provided the penalty is appropriately inflated.