[Paper Review] Doubly Robust Bayesian Inference for Non-Stationary Streaming Data with $β$-Divergences

Jeremias Knoblauch, Jack Jewson+1 more|arXiv (Cornell University)|Jun 6, 2018

Statistical Methods and Inference5 citations

TL;DR

This paper introduces the first robust Bayesian Online Changepoint Detection (BOCPD) algorithm using β-divergences, enabling doubly robust inference for non-stationary streaming data with linear time and constant space complexity. It achieves state-of-the-art performance by reducing false positive changepoint detections from over 90% to 0% on real-world data through principled β-parameter optimization and scalable Structural Variational Inference.

ABSTRACT

We present the very first robust Bayesian Online Changepoint Detection algorithm through General Bayesian Inference (GBI) with $β$-divergences. The resulting inference procedure is doubly robust for both the parameter and the changepoint (CP) posterior, with linear time and constant space complexity. We provide a construction for exponential models and demonstrate it on the Bayesian Linear Regression model. In so doing, we make two additional contributions: Firstly, we make GBI scalable using Structural Variational approximations that are exact as $β o 0$. Secondly, we give a principled way of choosing the divergence parameter $β$ by minimizing expected predictive loss on-line. Reducing False Discovery Rates of CPs from more than 90% to 0% on real world data, this offers the state of the art.

Motivation & Objective

• To develop the first robust Bayesian Online Changepoint Detection (BOCPD) algorithm for non-stationary streaming data.
• To enable doubly robust inference for both model parameters and changepoints under model misspecification and outliers.
• To provide a scalable inference framework for General Bayesian Inference (GBI) using Structural Variational Inference that is exact as β → 0.
• To establish a principled, on-line method for initializing and optimizing the β-divergence parameter to balance robustness and predictive accuracy.

Proposed method

• Proposes a novel BOCPD framework based on General Bayesian Inference (GBI) using β-divergences instead of Kullback-Leibler divergence to achieve robustness.
• Employs Structural Variational Inference (SVI) with exact posterior approximation in the limit as β → 0, enabling scalable inference with linear time and constant space complexity.
• Introduces a variance-reduced Stochastic Gradient Descent (SGD) scheme to efficiently optimize the SVI objective during online processing.
• Develops an on-line β-optimization strategy that minimizes expected predictive loss using bounded absolute loss, allowing dynamic adjustment of robustness.
• Applies the method to Bayesian Linear Regression (BLR) and multivariate vector autoregressive (VAR) models, with preprocessing for numerical stability.
• Uses a run-length model with prior h(rt) = 0.001 and hyperparameters µ₀ = 0, Σ₀ = 20I, a₀ = 1, b₀ = 25 to initialize the inference process.

Experimental results

Research questions

• RQ1 Can β-divergence-based General Bayesian Inference enable robust, scalable, and doubly robust changepoint detection in streaming data?
• RQ2 How can Structural Variational Inference be made both efficient and exact in the β → 0 limit for GBI in online settings?
• RQ3 What is a principled, on-line method for selecting and adapting the β parameter to balance robustness and predictive performance?
• RQ4 Can the proposed method reduce false positive changepoint detections in real-world data where standard BOCPD fails due to outliers?
• RQ5 How does the performance of the robust BOCPD compare to non-robust BOCPD in terms of predictive accuracy and changepoint detection under model misspecification?

Key findings

• The robust BOCPD reduces false positive changepoint detections from over 90% to 0% on real-world air pollution data, significantly outperforming standard BOCPD.
• The method maintains high predictive accuracy even when the data contains outliers, as demonstrated on multivariate time series with injected t-distributed noise.
• The on-line β-optimization procedure stabilizes quickly, with βp changing by less than 0.002 and βrld by less than 0.015 over 4,000 observations in the well-log data.
• The Structural Variational Inference approach achieves near-perfect fit to the true posterior as β → 0, enabling exact inference in the limit.
• The optimization of βp is constrained by numerical stability to ε = 1e-10, but the algorithm still drives βp toward less robust values, indicating strong on-line adaptability.
• Despite using fewer models (lag lengths 1–3) than the non-robust baseline (with lags 1,5,6,7), the robust model achieves better predictive performance and fewer false positives.