[Paper Review] Log-based Anomaly Detection based on EVT Theory with feedback

Jinyang Liu, Junjie Huang+7 more|arXiv (Cornell University)|Jun 8, 2023

Software System Performance and Reliability4 citations

TL;DR

ScaleAD is a lightweight, adaptive log-based anomaly detection framework that uses a Trie-based Detection Agent (TDA) for real-time, low-resource anomaly detection in cloud systems. By incorporating feedback from human experts or large language models (LLMs), it maintains high accuracy even as logs evolve, outperforming baselines with 2× to 10× faster inference and 5% to 41% lower memory usage.

ABSTRACT

System logs play a critical role in maintaining the reliability of software systems. Fruitful studies have explored automatic log-based anomaly detection and achieved notable accuracy on benchmark datasets. However, when applied to large-scale cloud systems, these solutions face limitations due to high resource consumption and lack of adaptability to evolving logs. In this paper, we present an accurate, lightweight, and adaptive log-based anomaly detection framework, referred to as SeaLog. Our method introduces a Trie-based Detection Agent (TDA) that employs a lightweight, dynamically-growing trie structure for real-time anomaly detection. To enhance TDA's accuracy in response to evolving log data, we enable it to receive feedback from experts. Interestingly, our findings suggest that contemporary large language models, such as ChatGPT, can provide feedback with a level of consistency comparable to human experts, which can potentially reduce manual verification efforts. We extensively evaluate SeaLog on two public datasets and an industrial dataset. The results show that SeaLog outperforms all baseline methods in terms of effectiveness, runs 2X to 10X faster and only consumes 5% to 41% of the memory resource.

Motivation & Objective

• Address the limitations of existing log-based anomaly detection methods in large-scale cloud systems, particularly high resource consumption and poor adaptability to evolving logs.
• Design a system that maintains high detection accuracy while operating under strict resource constraints (e.g., 1 CPU core, 200MB memory per instance).
• Enable continuous adaptation to evolving log patterns—common in cloud environments due to frequent software updates—without retraining from scratch.
• Explore the feasibility of using large language models (e.g., ChatGPT) as expert feedback sources to reduce manual labeling effort.
• Develop a scalable and efficient solution suitable for deployment at scale across thousands of cloud instances.

Proposed method

• Propose a Trie-based Detection Agent (TDA) that uses a lightweight, dynamically growing trie structure to efficiently index and detect anomalous log patterns in real time.
• Apply Extreme Value Theory (EVT) to model the tail distribution of log frequencies, enabling robust detection of rare or anomalous log events.
• Integrate a feedback loop where expert-labeled anomalies are used to refine the TDA’s detection logic, improving accuracy over time.
• Leverage large language models (LLMs) as automated expert feedback sources to assess log semantic similarity and validate anomaly predictions.
• Design the system for streaming log processing, enabling low-latency, in-situ detection directly on cloud instances without relying on centralized computation.
• Optimize the TDA’s memory and computation usage through incremental trie updates and EVT-based thresholding, ensuring minimal resource overhead.

Experimental results

Research questions

• RQ1 Can a lightweight, in-instance anomaly detection system achieve high accuracy in real-world cloud environments with limited CPU and memory?
• RQ2 How effective is the integration of expert feedback—especially from LLMs—compared to human experts in improving anomaly detection accuracy over time?
• RQ3 To what extent can EVT-based modeling of log frequency tails enhance detection of rare or novel anomalies in evolving log streams?
• RQ4 How does the proposed TDA-based approach scale in terms of speed and memory efficiency compared to existing ML and DL-based log anomaly detection methods?
• RQ5 Does the system maintain high performance when log patterns evolve significantly over time, such as during software release cycles?

Key findings

• ScaleAD achieves F1 scores between 0.908 and 0.990 across two public datasets and an industrial dataset from Huawei Cloud, demonstrating high detection accuracy.
• The framework runs 2× to 10× faster than baseline methods, significantly improving inference speed in real-time log processing.
• ScaleAD consumes only 5% to 41% of the memory used by existing methods, making it suitable for deployment on resource-constrained cloud instances.
• Feedback from LLMs like ChatGPT achieves anomaly detection consistency comparable to human experts, reducing the need for manual verification.
• The system maintains high accuracy even when log patterns evolve—on average, 14.5% of logs are new per month in a 20-microservice system, with 85% of log pairs showing low semantic similarity.
• The combination of TDA and feedback-driven adaptation enables sustained performance on evolving logs, outperforming both static models and purely unsupervised deep learning approaches.