[논문 리뷰] LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding

Shiwen Shan, Yintong Huo외 4명|arXiv (Cornell University)|2026. 03. 21.

Software System Performance and Reliability인용 수 0

한 줄 요약

LogFold은 골격(skeleton)-인식 구조화 토큰 분석과 유형 인식 하이브리드 인코딩 파이프라인을 도입하여 로그를 압축하고, 16개의 공개 데이터셋에서 최첨단 베이스라인을 능가합니다.

ABSTRACT

Logs are essential for diagnosing failures and conducting retrospective studies, leading many software organizations to retain log messages for a long time. Nevertheless, the volume of generated log data grows rapidly as software systems grow, necessitating an effective compression method. Apart from general-purpose compressors (e.g., Gzip, Bzip2), many recent studies developed log-specific compression algorithms, but they offer suboptimal performance because of (1) overlooking redundancies within certain complex tokens, and (2) lacking a fine-grained encoding strategy for diverse token types. This work uncovers a new redundancy pattern in structured tokens and proposes a new type-aware encoding strategy to improve log compression. Building on this insight, we introduce LogFold, a novel log compression method consisting of four components: a token analyzer to classifies tokens as structured, unstructured, or static types; a processor that mines recurring patterns within structured tokens based on their delimiter skeletons; a hybrid encoder that tailors data representation according to token types; and a packer that compresses the output into an archive file. Extensive experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s. Ablation studies further indicate the importance of each component. We also conduct sensitivity analyses to verify LogFold's robustness and stability across various internal settings.

연구 동기 및 목표

• 로그 내 구조화된 토큰의 중복성을 식별하여 압축을 개선한다.
• 효율적 로그 압축을 위한 네 구성 파이프라인(token analyzer, structured token processor, hybrid encoder, packer)을 제안한다.
• 숫자, 문자열, 혼합 타입 토큰에 맞춘 유형 인식 인코딩 전략을 개발한다.
• 다양한 공개 로그 데이터셋에서 LogFold를 평가하고 최첨단 로그 압축기 및 범용 압축기와 비교한다.

제안 방법

• Token Analyzer는 각 로그 항목에 대해 토큰을 구조화된지, 비구조화된지, 또는 정적(Static)으로 분류한다.
• Structured Token Processor는 Delimiter Skeleton-aware Grouping 및 Pattern Mining을 수행하여 토큰 내 중복성을 추출한다.
• Hybrid Encoder는 토큰 타입에 맞춘 최적화된 숫자 인코딩, 사전 인코딩, 혼합 타입 인코딩을 적용한다.
• Packer는 중간 출력을 모아 일반 목적의 압축기를 적용하여 최종 아카이브를 생성한다.
• Decompressor는 손실 없는 복구를 보장하기 위해 파이프라인을 역으로 실행한다.

Figure 1. The general log compressor paradigm.

실험 결과

연구 질문

• RQ1 RQ1: How well does LogFold improve log compression?
• RQ2 RQ2: How do different components contribute to LogFold’s effectiveness?
• RQ3 RQ3: How sensitive is LogFold to its internal parameter settings?
• RQ4 RQ4: How generalizable is LogFold across different zip tools with different compression levels?
• RQ5 RQ5: How does LogFold perform in log decompression?

주요 결과

• LogFold achieves an average compression ratio improvement of 11.11% over state-of-the-art baselines on 16 public datasets.
• LogFold achieves a compression speed of 9.842 MB/s.
• LogFold outperforms nine baseline compressors across the evaluation datasets and attains the best compression on 12 of 16 datasets.
• Ablation studies show the contribution of each component (token analyzer, structured token processor, hybrid encoder, packer).
• Sensitivity analyses confirm LogFold’s robustness and stability across internal settings.

Figure 2. Examples of structured tokens.