[Paper Review] LogStamp: Automatic Online Log Parsing Based on Sequence Labelling

Shimin Tao, Weibin Meng+7 more|arXiv (Cornell University)|Aug 10, 2022

Software System Performance and Reliability4 citations

TL;DR

LogStamp is an online log parsing method that frames log parsing as a sequence labeling task using BERT and a tagger, enabling high-accuracy parsing with minimal training data. It achieves an average accuracy of 0.956 using only 10% of the training data, making it effective for real-time analysis of new or evolving system logs.

ABSTRACT

Logs are one of the most critical data for service management. It contains rich runtime information for both services and users. Since size of logs are often enormous in size and have free handwritten constructions, a typical log-based analysis needs to parse logs into structured format first. However, we observe that most existing log parsing methods cannot parse logs online, which is essential for online services. In this paper, we present an automatic online log parsing method, name as LogStamp. We extensively evaluate LogStamp on five public datasets to demonstrate the effectiveness of our proposed method. The experiments show that our proposed method can achieve high accuracy with only a small portion of the training set. For example, it can achieve an average accuracy of 0.956 when using only 10% of the data training.

Motivation & Objective

• Address the challenge of online log parsing in dynamic production environments where new log types emerge continuously.
• Overcome the limitation of existing log parsing methods that require large volumes of historical log data to achieve high accuracy.
• Enable real-time, accurate log parsing for new services with limited historical logs by leveraging sequence labeling and pre-trained language models.
• Ensure robust performance across diverse log datasets with varying structures and vocabularies, especially under low-data regimes.

Proposed method

• Transform log parsing into a sequence labeling problem, where each token in a log is labeled as either part of a template or a variable placeholder.
• Utilize BERT to encode contextual representations of log sequences, capturing semantic and syntactic patterns for improved template identification.
• Train a sequence tagger (e.g., LSTM) on the BERT-encoded embeddings to predict labels for each token in a log line.
• Support online parsing by processing logs sequentially, enabling real-time template extraction without requiring batch processing.
• Fine-tune BERT on log-specific data to improve alignment with log-specific language patterns, though experiments show limited gains in this setting.
• Use a two-stage process: offline pre-training on labeled log sequences to learn template patterns, followed by online inference for real-time parsing.

Experimental results

Research questions

• RQ1 Can LogStamp achieve high accuracy in online log parsing with minimal training data?
• RQ2 How stable is LogStamp’s performance across different scales of training data, especially with only 10% of the dataset?
• RQ3 What is the contribution of BERT and the sequence tagger (e.g., LSTM, CNN) to the overall parsing accuracy?
• RQ4 How does LogStamp compare to existing log parsing methods in terms of accuracy and scalability on public datasets?

Key findings

• LogStamp achieves an average parsing accuracy of 0.956 when trained on only 10% of the training data across five public datasets.
• The model maintains high stability and accuracy across all five datasets, with online accuracy consistently above 0.95 even with small training sets.
• LSTM-based taggers outperform GCN, RNN, and CNN in sequence labeling, achieving the highest accuracy on all datasets.
• BERT-base, BERT-small, and BERT-tiny yield comparable performance, indicating that model size does not significantly affect accuracy in this setup.
• The model demonstrates strong generalization, with online accuracy on Hadoop and BGL datasets exceeding 0.997 even with minimal data.
• Fine-tuning BERT on log data did not yield consistent improvements in clustering or parsing performance, suggesting that pre-trained BERT already captures sufficient log semantics for the task.