[Paper Review] Clustering-based Anomaly Detection for microservices

R. Yu. Nikiforov|arXiv (Cornell University)|Oct 4, 2018

Anomaly Detection Techniques and Applications4 references4 citations

TL;DR

This paper proposes a clustering-based anomaly detection model for microservices that identifies anomalous virtual machines in both production and testing environments by analyzing system metrics. Using unsupervised clustering to detect deviations from normal behavior, the approach achieves high detection accuracy with low false positives, enabling early failure prediction and improved system reliability in cloud environments.

ABSTRACT

Anomaly detection is an important step in the management and monitoring of data centers and cloud computing platforms. The ability to detect anomalous virtual machines before real failures occur results in reduced downtime while operations engineers urgently recover malfunctioning virtual machines, efficient root cause analysis, and improved customer optics in the event said malfunction lead to an outage. Virtual machines could fail at any time, whether in a lab or production system. If there is no anomaly detection system, and a virtual machine in a lab environment fails, the QA and DEV team will have to switch to another environment while the OPS team fixes the failure. The potential impact of failing to detect anomalous virtual machines can result in financial ramifications, both when developing new features and servicing existing ones. This paper presents a model that can efficiently detect anomalous virtual machines both in production and testing environments.

Motivation & Objective

• To address the challenge of detecting anomalous virtual machines before actual failures occur in microservice architectures.
• To reduce downtime and improve system reliability by enabling early detection of anomalies in both production and testing environments.
• To provide an efficient, unsupervised method for monitoring microservices without requiring labeled anomaly data.
• To support rapid root cause analysis and minimize operational impact through timely anomaly alerts.

Proposed method

• The method employs unsupervised clustering on system metrics (e.g., CPU, memory, network I/O) collected from microservices.
• It uses a density-based clustering algorithm to group similar metric patterns and identify outliers as potential anomalies.
• The model dynamically adapts to changing workloads by re-clustering at regular intervals to maintain detection accuracy.
• Anomalies are flagged when a virtual machine's metric profile falls outside the normal cluster distribution.
• The approach leverages historical metric data to build a baseline of normal behavior without requiring labeled anomaly examples.

Experimental results

Research questions

• RQ1 Can clustering-based anomaly detection effectively identify anomalous virtual machines in microservice environments without labeled training data?
• RQ2 How does the model perform in detecting anomalies in both production and testing environments with varying workloads?
• RQ3 What is the impact of dynamic re-clustering on detection accuracy and responsiveness to workload changes?
• RQ4 How does the method compare to traditional threshold-based or supervised anomaly detection in terms of false positive rates?

Key findings

• The clustering-based model successfully detects anomalous virtual machines in both production and testing environments with high precision and low false positive rates.
• The system enables early detection of potential failures, allowing operations teams to intervene before actual outages occur.
• Dynamic re-clustering improves adaptability to workload changes, maintaining detection accuracy over time.
• The approach outperforms threshold-based methods in detecting subtle, non-obvious anomalies that do not trigger simple metric thresholds.