[Paper Review] Real-Time Anomaly Detection for Advanced Manufacturing: Improving on Twitter's State of the Art

Caitríona M. Ryan, Andrew Parnell+1 more|arXiv (Cornell University)|Nov 13, 2019

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes R-ESD, a statistically principled, real-time anomaly detection algorithm for streaming time series data in advanced manufacturing. By recursively updating the generalized extreme studentized deviate (ESD) test statistic using rolling windows and mean-based standardization, R-ESD enables low-memory, real-time detection that outperforms Twitter’s SH-ESD and other benchmarks in precision, recall, and early anomaly detection on industrial and benchmark datasets.

ABSTRACT

The detection of anomalies in real time is paramount to maintain performance and efficiency across a wide range of applications including web services and smart manufacturing. This paper presents a novel algorithm to detect anomalies in streaming time series data via statistical learning. We adapt the generalised extreme studentised deviate test [1] to streaming data by using a sliding window approach. This is made computationally feasible by recursive updates of the Grubbs test statistic [2]. Moreover, a priority queue [3] is employed to reduce memory requirements, where subsets of the required data streaming window are maintained in the algorithm rather than the full list. Our method is statistically principled. It is suitable for streaming data and it outperforms the AnomalyDetection software package, recently released by Twitter Inc. (Twitter) [4] and used by multiple teams at Twitter as their state of the art on a daily basis [5]. The methodology is demonstrated using an example of unlabelled data from the Twitter AnomalyDetection GitHub repository and using a real manufacturing example with labelled anomalies.

Motivation & Objective

• To develop a real-time, low-memory anomaly detection method for streaming time series data in advanced manufacturing.
• To address statistical and computational limitations of Twitter’s SH-ESD, particularly its use of median and MAD for studentization and non-overlapping windows.
• To enable true streaming detection via recursive updates of the ESD test statistic in rolling windows.
• To improve detection accuracy and early warning capability for anomalies in industrial sensor data.
• To outperform existing state-of-the-art methods, including SH-ESD, EGADS, and DeepADVote, on benchmark and real-world datasets.

Proposed method

• The R-ESD algorithm uses time series decomposition to estimate and remove trend and seasonality in an initial phase.
• It applies the generalized ESD test to residuals within a sliding window of streaming data using recursive updates of the test statistic.
• The test statistic is updated recursively using the mean and standard deviation for proper studentization, aligning with the theoretical distribution of the ESD test.
• The period for seasonality is estimated using the periodogram function in the TSA package via Fourier transformation.
• Anomalies are detected by comparing the test statistic to critical values derived from the ESD distribution, with multiple outliers tested per window.
• The method supports real-time deployment by minimizing memory usage and enabling constant-time updates per data point.

Experimental results

Research questions

• RQ1 Can a statistically principled version of the generalized ESD test be adapted for real-time streaming time series data?
• RQ2 Does replacing median and MAD with mean and standard deviation in studentization improve anomaly detection performance and reduce type I error?
• RQ3 Can recursive updates of the ESD test statistic enable low-memory, real-time anomaly detection in rolling windows?
• RQ4 How does R-ESD compare to Twitter’s SH-ESD and other state-of-the-art methods in detecting anomalies in industrial and benchmark datasets?
• RQ5 Can R-ESD detect subtle, gradual anomalies earlier than existing methods, particularly in manufacturing contexts?

Key findings

• R-ESD achieved a precision of 0.004 and recall of 0.25 on the Numenta machine temperature dataset, correctly identifying the first anomaly not detected by SH-ESD.
• SH-ESD failed to detect any anomalies on the same dataset, recording both precision and recall as 0.0.
• On the Yahoo EGADS A3 benchmark, R-ESD achieved an F1-score greater than 0.75, outperforming EGADS (F1 < 0.7) and DeepADVote (25th percentile F1 = 0.3).
• R-ESD required only 0.0005 seconds per window to process 20,000 windows, demonstrating high computational efficiency for real-time streaming.
• The algorithm correctly detected a gradual temperature decline leading to catastrophic failure, demonstrating early warning capability crucial for industrial maintenance.
• R-ESD’s use of mean-based studentization is statistically principled and avoids the heavy-tailed distribution issues associated with median and MAD in SH-ESD.