[Paper Review] Query Log Compression for Workload Analytics

Ting Xie, Oliver Kennedy+1 more|arXiv (Cornell University)|Sep 2, 2018

Advanced Database Systems and Queries37 references3 citations

TL;DR

This paper proposes LogR, a lossy query log compression scheme that uses pattern-based encoding and clustering to efficiently summarize database workloads for analytics. By introducing Reproduction Error as a computationally efficient proxy for fidelity, LogR achieves faster, more accurate workload summarization than state-of-the-art methods while enabling both machine and human interpretation of compressed logs.

ABSTRACT

Analyzing database access logs is a key part of performance tuning, intrusion detection, benchmark development, and many other database administration tasks. Unfortunately, it is common for production databases to deal with millions or even more queries each day, so these logs must be summarized before they can be used. Designing an appropriate summary encoding requires trading off between conciseness and information content. For example: simple workload sampling may miss rare, but high impact queries. In this paper, we present LogR, a lossy log compression scheme suitable use for many automated log analytics tools, as well as for human inspection. We formalize and analyze the space/fidelity trade-off in the context of a broader family of "pattern" and "pattern mixture" log encodings to which LogR belongs. We show through a series of experiments that LogR compressed encodings can be created efficiently, come with provable information-theoretic bounds on their accuracy, and outperform state-of-art log summarization strategies.

Motivation & Objective

• To address the challenge of analyzing massive-scale database query logs that are too large for direct processing.
• To design a lossy compression scheme that preserves critical statistical properties of workloads, especially rare but high-impact queries.
• To develop a computationally efficient measure of encoding fidelity that correlates strongly with classical metrics like Ambiguity and Deviation.
• To reduce the search space for optimal log summaries by using clustering-based pattern mixture encodings.
• To outperform existing state-of-the-art log summarization techniques in both accuracy and efficiency.

Proposed method

• LogR represents queries as bit-vector feature vectors based on structural elements like SELECT, FROM, and WHERE clauses.
• It introduces a family of pattern encodings that map frequently co-occurring structural elements (patterns) to their frequencies in the log.
• To address the intractability of optimizing over all possible encodings, LogR uses clustering to partition logs into groups, enabling pattern mixture encoding.
• It proposes a simplified approach called naive mixture encoding, where each cluster is compressed independently using its own pattern encoding.
• It introduces Reproduction Error as a practical, efficiently computable measure of encoding fidelity that closely tracks theoretical measures like Ambiguity and Deviation.
• The method avoids expensive inference by focusing on order-independent aggregate statistics, making it suitable for performance tuning, intrusion detection, and query recommendation.

Experimental results

Research questions

• RQ1 Can a lossy compression scheme be designed that balances compactness and fidelity in query log summarization while remaining efficient to compute?
• RQ2 Is Reproduction Error a reliable and efficient proxy for classical fidelity measures like Ambiguity and Deviation?
• RQ3 Can clustering-based pattern mixture encodings significantly reduce the search space for optimal log summaries without sacrificing accuracy?
• RQ4 How does LogR compare in performance and accuracy to state-of-the-art pattern-based summarization techniques?
• RQ5 Can the resulting compressed summaries be both machine-efficient and human-interpretable for database administration tasks?

Key findings

• LogR achieves faster compression and higher precision than state-of-the-art pattern encoding algorithms in workload summarization tasks.
• Reproduction Error is highly correlated with classical fidelity measures, making it a reliable and efficient alternative for evaluating log encodings.
• Naive mixture encodings, which compress each cluster independently, produce results competitive with more complex optimization techniques.
• The clustering-based approach to pattern mixture encoding significantly reduces the search space for optimal encodings, enabling scalable and efficient summary construction.
• LogR-compressed logs preserve critical workload statistics, including rare queries that may disproportionately affect performance, avoiding the pitfalls of simple sampling.
• The compressed summaries are both machine-efficient and human-interpretable, supporting both automated analytics and expert inspection.