[Paper Review] Anomaly Detection in Big Data

Chandresh Kumar Maurya|arXiv (Cornell University)|Mar 3, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This Ph.D. thesis proposes novel algorithms—PAGMEAN, ASPGD, DSCIL, and CILSD—for anomaly detection in big data, leveraging cost-sensitive learning, stochastic optimization, and distributed frameworks to handle high-dimensionality, sparsity, and scalability. The key contribution is improved performance in G-mean and robustness in distributed settings, especially on real-world and benchmark datasets including KDDCup 2008.

ABSTRACT

Anomaly is defined as a state of the system that do not conform to the normal behavior. For example, the emission of neutrons in a nuclear reactor channel above the specified threshold is an anomaly. Big data refers to the data set that is \emph{high volume, streaming, heterogeneous, distributed} and often \emph{sparse}. Big data is not uncommon these days. For example, as per Internet live stats, the number of tweets posted per day has gone above 500 millions. Due to data explosion in data laden domains, traditional anomaly detection techniques developed for small data sets scale poorly on large-scale data sets. Therefore, we take an alternative approach to tackle anomaly detection in big data. Essentially, there are two ways to scale anomaly detection in big data. The first is based on the \emph{online} learning and the second is based on the \emph{distributed} learning. Our aim in the thesis is to tackle big data problems while detecting anomaly efficiently. To that end, we first take \emph{streaming} issue of the big data and propose Passive-Aggressive GMEAN (PAGMEAN) algorithms. Although, online learning algorithm can scale well over large number of data points and dimensions, they can not process data when it is distributed at multiple locations; which is quite common these days. Therefore, we propose anomaly detection algorithm which is inherently distributed using ADMM. Finally, we present a case study on anomaly detection in nuclear power plant data.

Motivation & Objective

• Address the challenge of detecting anomalies in high-dimensional, sparse, and distributed big data where traditional supervised methods fail due to lack of labels.
• Develop scalable, efficient, and robust anomaly detection algorithms that handle class imbalance and streaming data characteristics.
• Enable real-world deployment through distributed and online learning frameworks with minimal hyperparameter tuning.
• Evaluate performance on both benchmark and real-world datasets, including nuclear power plant and KDDCup 2008 data, to demonstrate practical applicability.

Proposed method

• Proposes PAGMEAN by integrating a surrogate loss function within the Passive-Aggressive (PA) framework to handle non-convexity in cost-sensitive learning, improving G-mean performance.
• Introduces ASPGD, a stochastic proximal learning algorithm with Nesterov acceleration and a cost-sensitive smooth hinge loss for streaming, high-dimensional data.
• Develops DSCIL using a distributed ADMM framework with L2-regularized cost-sensitive loss, employing L-BFGS (L-DSCIL) and Random Coordinate Descent (R-DSCIL) for optimization.
• Proposes CILSD as a parameter-free, distributed FISTA-like algorithm that avoids learning rate tuning and uses adaptive step sizes via Lipschitz constant.
• Employs SVDD as an unsupervised baseline for real-world unlabeled data, particularly in nuclear power plant monitoring, to validate effectiveness without labels.
• Uses performance metrics including G-mean, mistake rate, F-measure, and convergence speed across multiple datasets and hardware configurations (multi-core systems).

Experimental results

Research questions

• RQ1 How can anomaly detection be effectively performed in high-dimensional, sparse, and distributed big data environments with minimal labeling?
• RQ2 Can cost-sensitive learning improve G-mean performance in imbalanced anomaly detection tasks while maintaining low mistake rates?
• RQ3 How does Nesterov acceleration affect performance in online anomaly detection for streaming data?
• RQ4 What are the trade-offs between convergence speed, training time, and G-mean in distributed optimization frameworks like ADMM and FISTA?
• RQ5 Can parameter-free distributed algorithms like CILSD outperform tuned counterparts in real-world anomaly detection scenarios?

Key findings

• PAGMEAN outperforms its parent algorithms PA and CSOC in terms of G-mean across all benchmark datasets, though it shows higher mistake rates.
• ASPGD achieves better G-mean and F-measure than CSFSOL on multiple benchmark datasets, though acceleration does not consistently improve performance.
• L-DSCIL converges faster and achieves G-mean close to centralized solutions, while R-DSCIL shows variable performance due to random coordinate updates.
• R-DSCIL demonstrates practical speedup and scalability on multi-core systems, with training time decreasing logarithmically with increasing cores.
• CILSD achieves high G-mean without tuning the learning rate, converging faster than DSCIL and outperforming or matching centralized algorithms on most datasets.
• On the KDDCup 2008 dataset, both R-DSCIL and CILSD show strong potential for real-world distributed anomaly detection, with CILSD showing faster convergence and stable performance.