[Paper Review] Self-Supervised Log Parsing

Sasho Nedelkoski, Jasmin Bogatinovski+3 more|arXiv (Cornell University)|Mar 17, 2020

Software System Performance and Reliability20 references4 citations

TL;DR

This paper proposes NuLog, a self-supervised log parsing method that formulates log parsing as masked language modeling (MLM) using a transformer-based architecture. By predicting masked tokens, NuLog extracts both structured log templates and dense vector embeddings, achieving 99% average parsing accuracy and outperforming 12 prior methods on 10 real-world datasets while enabling downstream anomaly detection with F1 scores up to 0.999.

ABSTRACT

Logs are extensively used during the development and maintenance of software systems. They collect runtime events and allow tracking of code execution, which enables a variety of critical tasks such as troubleshooting and fault detection. However, large-scale software systems generate massive volumes of semi-structured log records, posing a major challenge for automated analysis. Parsing semi-structured records with free-form text log messages into structured templates is the first and crucial step that enables further analysis. Existing approaches rely on log-specific heuristics or manual rule extraction. These are often specialized in parsing certain log types, and thus, limit performance scores and generalization. We propose a novel parsing technique called NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling (MLM). In the process of parsing, the model extracts summarizations from the logs in the form of a vector embedding. This allows the coupling of the MLM as pre-training with a downstream anomaly detection task. We evaluate the parsing performance of NuLog on 10 real-world log datasets and compare the results with 12 parsing techniques. The results show that NuLog outperforms existing methods in parsing accuracy with an average of 99% and achieves the lowest edit distance to the ground truth templates. Additionally, two case studies are conducted to demonstrate the ability of the approach for log-based anomaly detection in both supervised and unsupervised scenario. The results show that NuLog can be successfully used to support troubleshooting tasks. The implementation is available at https://github.com/nulog/nulog.

Motivation & Objective

• To address the limitations of existing log parsing methods that rely on manual heuristics or domain-specific rules.
• To develop a generalizable, low-human-intervention log parsing technique that works across diverse IT systems.
• To enable the use of learned log representations for downstream tasks such as anomaly detection.
• To evaluate the effectiveness of self-supervised learning in log parsing and its transferability to anomaly detection.

Proposed method

• Formulates log parsing as a masked language modeling (MLM) task using a transformer-based model.
• Uses the [CLS] token embedding to generate a dense vector representation of the log message context.
• Predicts masked tokens to distinguish between constant log template parts (correct prediction) and variable parameters (incorrect prediction).
• Leverages pre-trained MLM representations as a prior for downstream anomaly detection tasks.
• Fine-tunes the model for supervised anomaly detection by replacing the final softmax layer with a binary classification head.
• Applies unsupervised anomaly detection by flagging log messages with a high proportion of low-confidence masked token predictions.

Experimental results

Research questions

• RQ1 Can self-supervised masked language modeling effectively parse semi-structured log messages into structured templates with minimal human intervention?
• RQ2 How does NuLog's parsing performance compare to existing log parsing techniques across diverse real-world datasets?
• RQ3 Can the learned log embeddings from the self-supervised parsing phase be effectively transferred to downstream anomaly detection tasks?
• RQ4 What is the performance of NuLog in both supervised and unsupervised anomaly detection scenarios?

Key findings

• NuLog achieves an average parsing accuracy of 99% across 10 real-world log datasets, outperforming 12 existing log parsing methods.
• NuLog achieves the lowest edit distance to ground truth templates, indicating high structural fidelity in parsing.
• In unsupervised anomaly detection on the BGL dataset, NuLog achieves an F1 score of 0.999, with precision of 1.000 and recall of 0.999.
• In supervised anomaly detection, fine-tuning for just two epochs yields an F1 score of 0.999, demonstrating strong transferability of learned representations.
• The model generalizes well across diverse systems, including cloud infrastructure, mobile OS, and application logs, with minimal domain-specific adaptation.