[Paper Review] A Two-stage Online Monitoring Procedure for High-Dimensional Data Streams

Jun Li|arXiv (Cornell University)|Dec 14, 2017

Advanced Statistical Process Monitoring9 references3 citations

TL;DR

This paper proposes a two-stage online monitoring procedure for high-dimensional data streams that simultaneously controls the in-control average run length (IC-ARL) and Type-I error rate (false discovery rate) at user-specified levels. By decoupling the control of false alarms from detection power, the method overcomes limitations of existing single-stage approaches that either lack global FDR control or force a trade-off between IC-ARL and detection sensitivity.

ABSTRACT

Advanced computing and data acquisition technologies have made possible the collection of high-dimensional data streams in many fields. Efficient online monitoring tools which can correctly identify any abnormal data stream for such data are highly sought after. However, most of the existing monitoring procedures directly apply the false discover rate (FDR) controlling procedure to the data at each time point, and the FDR at each time point (the point-wise FDR) is either specified by users or determined by the in-control (IC) average run length (ARL). If the point-wise FDR is specified by users, the resulting procedure lacks control of the global FDR and keeps users in the dark in terms of the IC-ARL. If the point-wise FDR is determined by the IC-ARL, the resulting procedure does not give users the flexibility to choose the number of false alarms (Type-I errors) they can tolerate when identifying abnormal data streams, which often makes the procedure too conservative. To address those limitations, we propose a two-stage monitoring procedure that can control both the IC-ARL and Type-I errors at the levels specified by users. As a result, the proposed procedure allows users to choose not only how often they expect any false alarms when all data streams are IC, but also how many false alarms they can tolerate when identifying abnormal data streams. With this extra flexibility, our proposed two-stage monitoring procedure is shown in the simulation study and real data analysis to outperform the exiting methods.

Motivation & Objective

• To address the lack of global false discovery rate (FDR) control in existing online monitoring procedures for high-dimensional data streams.
• To resolve the inflexibility of current methods that tie the point-wise FDR to the in-control average run length (IC-ARL), limiting user control over false alarm tolerance.
• To develop a monitoring scheme that allows users to independently specify desired levels of IC-ARL and Type-I error rate when identifying out-of-control data streams.
• To improve detection power and false alarm control in high-dimensional streaming data by introducing a two-stage decision framework.

Proposed method

• Proposes a two-stage monitoring strategy where the first stage computes p-values for each data stream using CUSUM statistics, and the second stage applies a false discovery rate (FDR) controlling procedure to these p-values.
• Introduces a novel two-stage decision rule that separates the control of in-control average run length (IC-ARL) from the control of Type-I error rate (FDR), enabling independent user specification of both.
• Uses a hierarchical testing framework where the global FDR is controlled by adjusting thresholds in the second stage, while the IC-ARL is maintained via calibration of the first-stage control limits.
• Employs a likelihood ratio-based approach to prove that the two-stage procedure achieves monotonicity in the likelihood ratio, ensuring valid FDR control under the null and alternative hypotheses.
• Derives the relationship between p-values and test statistics using the probability density functions under the null and alternative hypotheses, enabling transformation of the monitoring problem into a p-value space.
• Applies a p-value transformation to ensure that the FDR is controlled globally over time, not just pointwise at each time point, by leveraging the dependence structure between test statistics.

Experimental results

Research questions

• RQ1 Does controlling the point-wise FDR at each time point guarantee control of the global FDR over a time window, as commonly assumed in the literature?
• RQ2 Can a monitoring procedure simultaneously control both the in-control average run length (IC-ARL) and the Type-I error rate (FDR) at user-specified levels?
• RQ3 How does the two-stage procedure compare to single-stage methods in terms of detection power and false alarm control in high-dimensional streaming data?
• RQ4 What is the impact of user-specified IC-ARL and FDR levels on the performance of the monitoring procedure in real-world applications?

Key findings

• The paper demonstrates that controlling the point-wise FDR does not guarantee control of the global FDR, challenging a widely held assumption in the literature.
• The proposed two-stage procedure successfully controls both the IC-ARL and the global FDR at user-specified levels, offering greater flexibility than existing single-stage methods.
• Simulation studies show that the two-stage method outperforms existing procedures in terms of detection speed and false alarm control, especially when the number of data streams is large.
• The method maintains the desired IC-ARL while allowing users to choose their preferred level of Type-I error tolerance, avoiding the conservative trade-off inherent in methods that fix the point-wise FDR via IC-ARL.
• Real data analysis on network traffic and health monitoring data confirms the method’s superior performance in identifying out-of-control streams with fewer false alarms.
• Theoretical proof shows that the likelihood ratio of the test statistics is monotonic, which ensures the validity of the FDR control under the two-stage decision rule.