[Paper Review] Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

Zhuangbin Chen, Jinyang Liu+3 more|arXiv (Cornell University)|Jul 13, 2021

Software System Performance and Reliability57 references63 citations

TL;DR

This paper surveys and benchmarks six DL-based log anomaly detectors (four unsupervised, two supervised) using two public datasets (HDFS, BGL), releases an open-source toolkit, and analyzes accuracy, robustness, and efficiency, showing supervised methods generally outperform unsupervised ones, and log semantics improves performance.

ABSTRACT

Logs have been an imperative resource to ensure the reliability and continuity of many software systems, especially large-scale distributed systems. They faithfully record runtime information to facilitate system troubleshooting and behavior understanding. Due to the large scale and complexity of modern software systems, the volume of logs has reached an unprecedented level. Consequently, for log-based anomaly detection, conventional manual inspection methods or even traditional machine learning-based methods become impractical, which serve as a catalyst for the rapid development of deep learning-based solutions. However, there is currently a lack of rigorous comparison among the representative log-based anomaly detectors that resort to neural networks. Moreover, the re-implementation process demands non-trivial efforts, and bias can be easily introduced. To better understand the characteristics of different anomaly detectors, in this paper, we provide a comprehensive review and evaluation of five popular neural networks used by six state-of-the-art methods. Particularly, four of the selected methods are unsupervised, and the remaining two are supervised. These methods are evaluated with two publicly available log datasets, which contain nearly 16 million log messages and 0.4 million anomaly instances in total. We believe our work can serve as a basis in this field and contribute to future academic research and industrial applications.

Motivation & Objective

• Motivate the need for DL-based log anomaly detection due to scale and complexity of modern software systems.
• Provide a comprehensive comparison of representative neural-network–based detectors (five models across six methods).
• Offer an open-source toolkit to ease adoption and replication in industry and research.
• Benchmark DL-based detectors against traditional ML baselines across accuracy, robustness, and efficiency.

Proposed method

• Review and classify six DL-based log anomaly detectors (four unsupervised, two supervised).
• Formulate loss functions as forecasting, reconstruction, or supervised objectives to guide model learning.
• Apply log parsing and log embedding to convert semi-structured logs into sequences for DL models.
• Evaluate on two public datasets (HDFS and BGL) using precision, recall, and F1 as metrics.
• Release an open-source PyTorch-based toolkit implementing the six methods for reuse.

Experimental results

Research questions

• RQ1 How do DL-based log anomaly detectors compare in accuracy, robustness, and efficiency on standard log datasets?
• RQ2 What is the impact of incorporating logs’ semantic information on detector performance?
• RQ3 How do supervised versus unsupervised DL methods compare in practice for log anomaly detection?
• RQ4 What is the effect of anomaly ratio in training data on detector performance?

Key findings

• Supervised DL methods generally outperform unsupervised ones on both datasets (e.g., F1 around 0.97–0.98 for supervised methods).
• Unsupervised LSTM achieves the best F1 of 0.944 on HDFS and 0.961 on BGL in the non-semantics setting.
• Incorporating log semantics improves performance for most models; on BGL, best F1 with semantics reaches 0.989 for CNN and 0.967 for LSTM in unsupervised settings.
• Logs’ semantics particularly benefit Unsup. methods, including Autoencoder and LSTM, with notable gains (e.g., Autoencoder improves around 15% in F1 on some settings).
• Compared to traditional ML baselines, DL-based methods generally perform better, with several traditional methods ranking below DL approaches on both datasets.
• An open-source toolkit (deep-loglizer) is provided to facilitate reuse and replication of the six detectors.