[Paper Review] Online Anomaly Detection Systems Using Incremental Commute Time

Nguyen Lu Dang Khoa, Sanjay Chawla|arXiv (Cornell University)|Jul 20, 2011

Anomaly Detection Techniques and Applications21 references3 citations

TL;DR

This paper proposes two novel incremental methods for computing Commute Time Distance (CTD) in real time: one based on incremental eigen-decomposition of the graph Laplacian, and another using recursive hitting time approximation. Both enable online anomaly detection with constant-time updates per new data point, achieving sub-7ms average processing time on large graphs (300K+ nodes), and outperform batch methods in speed while maintaining high accuracy in detecting global and local anomalies.

ABSTRACT

Commute Time Distance (CTD) is a random walk based metric on graphs. CTD has found widespread applications in many domains including personalized search, collaborative filtering and making search engines robust against manipulation. Our interest is inspired by the use of CTD as a metric for anomaly detection. It has been shown that CTD can be used to simultaneously identify both global and local anomalies. Here we propose an accurate and efficient approximation for computing the CTD in an incremental fashion in order to facilitate real-time applications. An online anomaly detection algorithm is designed where the CTD of each new arriving data point to any point in the current graph can be estimated in constant time ensuring a real-time response. Moreover, the proposed approach can also be applied in many other applications that utilize commute time distance.

Motivation & Objective

• To enable real-time online anomaly detection using Commute Time Distance (CTD), which is typically too slow for incremental updates due to O(n³) eigen-decomposition cost.
• To address the challenge of efficiently computing CTD when new data points arrive dynamically, without recomputing from scratch.
• To design a scalable, constant-time method for estimating CTD in evolving graphs, suitable for streaming applications like network monitoring or sensor data.
• To compare two distinct incremental CTD approximation techniques—eigen-decomposition update and recursive hitting time estimation—for accuracy, speed, and robustness.
• To validate the proposed methods on synthetic and real-world datasets, including network traffic and environmental monitoring data.

Proposed method

• The first method incrementally updates the eigenvalues and eigenvectors of the graph Laplacian matrix using first-order perturbation theory, enabling efficient spectral computation in evolving graphs.
• The second method uses the recursive relationship between hitting time and commute time, allowing CTD estimation in constant time per new point via dynamic update of hitting times.
• Both methods avoid full re-computation of the Laplacian eigen-decomposition, reducing time complexity from O(n³) to near-linear or constant time per update.
• The online anomaly detection algorithm computes the CTD of each new point to all existing points in constant time, flagging points with high CTD as anomalies.
• The approach integrates min-max scaling for normalization and uses graph construction based on k-nearest neighbors or fixed neighborhood radius.
• The methods are evaluated using iLED (incremental eigen-decomposition-based) and iECT (incremental CTD via hitting time) variants, with performance compared to batch CTD computation.

Experimental results

Research questions

• RQ1 Can CTD be computed incrementally in constant time to support real-time anomaly detection in dynamic data streams?
• RQ2 How do incremental eigen-decomposition and recursive hitting time estimation compare in accuracy and efficiency for online CTD computation?
• RQ3 To what extent do the two proposed methods preserve the robustness of CTD in detecting both global and local anomalies?
• RQ4 Can the incremental CTD approach maintain high detection accuracy while reducing computational cost compared to batch methods?
• RQ5 What are the trade-offs between precision and recall in the two incremental methods, and can they be combined for improved performance?

Key findings

• The proposed incremental CTD methods achieve sub-7ms average processing time per new data point on graphs with over 300,000 nodes and one million edges, enabling real-time deployment.
• iECT (incremental CTD via hitting time) achieved 100% recall and 84.6% precision on the NICTA network dataset, outperforming iLED in recall while maintaining acceptable precision.
• iLED achieved 100% precision but only 66.7% recall on the synthetic dataset, indicating a tendency to underestimate CTD for anomalies.
• iECT had an average anomaly score 2% higher than the batch method, while iLED was 2% lower, suggesting iECT overestimates non-anomalies slightly.
• On the NICTA dataset, iLED had 27.3% recall but 100% precision, while iECT achieved 100% recall with 84.6% precision, confirming the complementary strengths of the two methods.
• The results show that iECT is faster and more effective at detecting anomalies, while iLED is more conservative, suggesting potential for hybrid integration to improve overall detection accuracy.