[Paper Review] Stein Variational Online Changepoint Detection with Applications to Hawkes Processes and Neural Networks

Gianluca Detommaso, Hanne Hoitzing+2 more|arXiv (Cornell University)|Jan 23, 2019

Point processes and geometric inequalities34 references5 citations

TL;DR

This paper proposes Stein Variational Online Changepoint Detection (SVOCD), a scalable Bayesian online method that extends traditional BOCPD beyond exponential family distributions by combining Bayesian online changepoint detection with the Stein variational Newton (SVN) method for efficient posterior inference. SVOCD enables accurate, real-time changepoint detection in complex models like Hawkes processes and Bayesian LSTMs, outperforming SMC-based methods in posterior estimation and reducing both false positive and false negative rates.

ABSTRACT

Bayesian online changepoint detection (BOCPD) (Adams & MacKay, 2007) offers a rigorous and viable way to identify changepoints in complex systems. In this work, we introduce a Stein variational online changepoint detection (SVOCD) method to provide a computationally tractable generalization of BOCPD beyond the exponential family of probability distributions. We integrate the recently developed Stein variational Newton (SVN) method (Detommaso et al., 2018) and BOCPD to offer a full online Bayesian treatment for a large number of situations with significant importance in practice. We apply the resulting method to two challenging and novel applications: Hawkes processes and long short-term memory (LSTM) neural networks. In both cases, we successfully demonstrate the efficacy of our method on real data.

Motivation & Objective

• To address the limitations of Bayesian online changepoint detection (BOCPD) in non-conjugate, non-exponential family models where conjugate priors fail.
• To develop a computationally efficient, online Bayesian inference framework that supports real-time changepoint detection in complex, high-dimensional models.
• To extend BOCPD to models lacking closed-form posteriors, such as Hawkes processes and Bayesian LSTMs, by leveraging advanced variational inference techniques.
• To demonstrate the method’s efficacy on real-world data, particularly in detecting structural shifts in financial time series and point process dynamics.
• To compare SVOCD’s performance against SMC-based approaches in terms of posterior approximation accuracy and changepoint detection reliability.

Proposed method

• Integrates Bayesian Online Changepoint Detection (BOCPD) with the Stein Variational Newton (SVN) method to enable online, particle-based posterior approximation.
• Uses a set of particles to represent the posterior distribution, which are iteratively updated via deterministic transport to approximate the true posterior as data arrives.
• Leverages the posterior from the previous time step as the initial particle distribution for the next update, enabling fast online inference with minimal re-computation.
• Applies SVN’s superior convergence and scalability over SVGD to handle high-dimensional parameter spaces common in neural networks and point processes.
• Employs a sequential updating mechanism where particles are adjusted with only a few iterations per new data point, ensuring low detection delay.
• Combines BOCPD with Bayesian LSTMs by training the LSTM using SVN to maintain uncertainty estimates and improve predictive performance.

Experimental results

Research questions

• RQ1 Can SVOCD provide accurate, real-time changepoint detection in models outside the exponential family, such as Hawkes processes?
• RQ2 How does SVOCD compare to SMC in approximating complex, non-Gaussian posterior distributions in online settings?
• RQ3 Can SVOCD effectively detect structural changes in time series generated by point processes with time-varying intensity?
• RQ4 Does incorporating Bayesian uncertainty via SVN improve prediction performance in LSTMs compared to deterministic training?
• RQ5 To what extent does SVOCD reduce false positive and false negative rates compared to existing BOCPD extensions?

Key findings

• SVOCD successfully detects changepoints in real Bitcoin price data, identifying the peak and subsequent crash in late 2017 with high temporal precision.
• On synthetic Hawkes process data, SVOCD accurately tracks posterior shifts across changepoints, while SMC fails to maintain proper posterior representation in non-Gaussian regimes.
• SVN-based inference converges faster and more accurately than SMC, as evidenced by lower mean squared error in covariance trace estimation across multiple particle counts.
• SVOCD achieves lower mean false positive and false negative rates than BOCPD combined with SMC, with statistical significance confirmed over 30 runs.
• The Bayesian LSTM trained via SVN produces more robust predictions than deterministic LSTMs, as the posterior mean outperforms the mode in capturing signal trends.
• The particle-based posterior approximation in SVOCD asymptotically converges to the true posterior as particle count increases, validating the theoretical foundation of the method.