[Paper Review] A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?

Zhihan Jiang, Jinyang Liu+7 more|arXiv (Cornell University)|Aug 21, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Loghub-2.0, a large-scale, real-world-inspired dataset of 14 annotated log collections (avg. 3.6M lines each), and conducts a rigorous evaluation of 15 state-of-the-art log parsers. It proposes a new template-level metric to address class imbalance and reveals critical performance gaps in existing parsers—especially for rare and parameter-intensive log templates—highlighting the need for improved design in real-world deployment.

ABSTRACT

Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.

Motivation & Objective

• To address the limitations of existing log parsing benchmarks like Loghub-2k, which suffer from small scale and poor representativeness of real-world log distributions.
• To develop a large-scale, realistic dataset (Loghub-2.0) that better reflects the scale and diversity of production log data.
• To propose a new evaluation protocol with a template-level metric that reduces sensitivity to imbalanced data distributions.
• To comprehensively assess the performance of state-of-the-art log parsers on rare and parameter-intensive log templates, which are critical for system diagnosis.
• To guide future research by identifying key shortcomings in current log parsers, especially in efficiency and handling of complex log patterns.

Proposed method

• Designed a stringent annotation framework with five experienced researchers to ensure high accuracy and consistency in labeling log templates and parameters across 14 real-world systems.
• Constructed Loghub-2.0, a new benchmark dataset comprising 14 log collections totaling over 50 million log lines, with each dataset reflecting real-world system characteristics.
• Proposed a new evaluation metric—template-level F1 score—that mitigates bias from class imbalance and provides a more accurate assessment of parser performance on rare templates.
• Implemented a unified benchmarking protocol that evaluates parsers across multiple dimensions: overall accuracy, performance on rare templates, performance on parameter-intensive templates, and efficiency.
• Conducted a large-scale re-evaluation of 15 state-of-the-art log parsers using Loghub-2.0, with hyperparameter tuning and consistent implementation across all tools.
• Analyzed parser behavior across different log characteristics, including frequency of occurrence and number of parameters, to uncover hidden performance issues.

Experimental results

Research questions

• RQ1 How do existing log parsers perform on large-scale, real-world log data compared to their reported results on Loghub-2k?
• RQ2 To what extent do current log parsers struggle with rare log templates and parameter-intensive log patterns, which are critical for system diagnosis?
• RQ3 How effective is the proposed template-level F1 metric in reducing bias from imbalanced data distributions in log parsing evaluation?
• RQ4 What are the key performance bottlenecks of current log parsers in large-scale production environments, particularly in terms of efficiency and accuracy on edge cases?
• RQ5 How can future log parsing techniques be improved to better handle rare and complex log patterns in real-world systems?

Key findings

• Existing log parsers, including state-of-the-art tools like Drain, Logram, and LogPPT, show significantly reduced performance on rare log templates—especially those with error or fatal severity—despite strong performance on Loghub-2k.
• Log parsers exhibit poor accuracy on parameter-intensive templates, with some achieving less than 50% F1 on templates with more than 10 parameters, indicating a major gap in handling complex log structures.
• The proposed template-level F1 metric effectively reduces sensitivity to class imbalance and reveals performance disparities that are masked by traditional message-level metrics.
• Many parsers fail to meet real-world efficiency requirements, with processing times exceeding acceptable thresholds for large-scale log streams, especially when handling high-velocity data.
• Semantic-based parsers show superior ability to distinguish parameters from templates but suffer from weak grouping capabilities due to lack of global context, suggesting a need to integrate statistical and semantic information.
• The study identifies that current log parsers are not optimized for production deployment, particularly in diagnosing rare and complex system events, calling for a paradigm shift in parser design.