[Paper Review] An Assessment of ChatGPT on Log Data

Priyanka Mudgal, Rita H. Wouhaybi|arXiv (Cornell University)|Sep 14, 2023

Software System Performance and ReliabilityComputer Science3 citations

TL;DR

This paper evaluates ChatGPT's capabilities across log parsing, log analytics, and log summarization using custom-designed prompts on real log datasets. It finds strong performance in log parsing and security information extraction, but significant limitations in API detection, anomaly identification, scalability due to token limits, and high latency—highlighting key challenges for real-time log processing applications.

ABSTRACT

Recent development of large language models (LLMs), such as ChatGPT has been widely applied to a wide range of software engineering tasks. Many papers have reported their analysis on the potential advantages and limitations of ChatGPT for writing code, summarization, text generation, etc. However, the analysis of the current state of ChatGPT for log processing has received little attention. Logs generated by large-scale software systems are complex and hard to understand. Despite their complexity, they provide crucial information for subject matter experts to understand the system status and diagnose problems of the systems. In this paper, we investigate the current capabilities of ChatGPT to perform several interesting tasks on log data, while also trying to identify its main shortcomings. Our findings show that the performance of the current version of ChatGPT for log processing is limited, with a lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role of LLMs in the log processing discipline and possible next steps to improve the current capabilities of ChatGPT and the future LLMs in this area. We believe our work can contribute to future academic research to address the identified issues.

Motivation & Objective

• To evaluate the current capabilities of ChatGPT in processing real-world log data across key software engineering tasks.
• To identify the limitations of ChatGPT in handling complex, unstructured log messages at scale.
• To investigate the feasibility of using zero-shot and few-shot prompting for log-specific tasks without fine-tuning.
• To assess the practical challenges of deploying LLMs like ChatGPT in real-time log analysis pipelines.
• To provide a foundation for future research on improving LLMs for log processing through structured evaluation and prompt design.

Proposed method

• Designed seven distinct prompts targeting specific log processing tasks: log parsing, API detection, error summarization, security threat detection, anomaly identification, event prediction, and log summarization.
• Applied prompts to real log datasets (e.g., Loghub) with raw, unstructured log messages to evaluate zero-shot performance.
• Evaluated responses across multiple dimensions: accuracy, completeness, consistency, and response time.
• Measured scalability by testing the maximum number of log messages processed per API call (limited to ~190–200 messages due to token constraints).
• Analyzed latency across different input sizes, observing response times from seconds to minutes.
• Assessed privacy risks by evaluating whether sensitive data (e.g., IPs, URLs, user IDs) was exposed or retained during processing.

Experimental results

Research questions

• RQ1 Can ChatGPT accurately extract log templates and variables from unstructured log messages without fine-tuning?
• RQ2 How effective is ChatGPT in detecting frequently called APIs, security threats, and system anomalies from raw logs?
• RQ3 What are the scalability limits of ChatGPT when processing large volumes of log data in a single prompt?
• RQ4 How does response latency vary with increasing log volume, and can it support real-time log analysis?
• RQ5 What are the privacy risks associated with sending log data to third-party LLM APIs like ChatGPT?

Key findings

• ChatGPT demonstrated strong performance in log parsing, successfully extracting templates and variables from raw log messages with high accuracy.
• The model showed consistent ability to detect and extract sensitive information such as IPs, URLs, and user IDs, indicating strong privacy and security analysis capabilities.
• Performance declined significantly in API detection and event prediction tasks, with multiple critical APIs and events being missed in the responses.
• Anomaly detection was inconsistent, with ChatGPT failing to identify known anomalies in several test cases, especially in complex or noisy log sequences.
• Response latency increased substantially with input size, ranging from seconds to minutes when processing larger log batches, limiting real-time applicability.
• The model’s token limit restricted processing to approximately 190–200 log messages per prompt, posing a major scalability barrier for industrial-scale log analysis.