[Paper Review] Decreasing log data of multi-tier services for effective request tracing

Bo Sang, Jianfeng Zhan+1 more|arXiv (Cornell University)|Mar 4, 2010

Software System Performance and Reliability2 references3 citations

TL;DR

This paper proposes a novel algorithm to reduce redundant log data in multi-tier service architectures by filtering out valueless causal paths during request tracing. By identifying and eliminating non-informative logs, the method reduces log volume by 84%, significantly improving scalability and efficiency in large-scale data centers without sacrificing debugging effectiveness.

ABSTRACT

Previous work shows request tracing systems help understand and debug the performance problems of multi-tier services. However, for large-scale data centers, more than hundreds of thousands of service instances provide online service at the same time. Previous work such as white-box or black box tracing systems will produce large amount of log data, which would be correlated into large quantities of causal paths for performance debugging. In this paper, we propose an innovative algorithm to eliminate valueless logs of multitiers services. Our experiment shows our method filters 84% valueless causal paths and is promising to be used in large-scale data centers.

Motivation & Objective

• To address the scalability challenge of request tracing in large-scale multi-tier services with hundreds of thousands of service instances.
• To reduce the volume of log data generated by traditional white-box and black-box tracing systems.
• To eliminate valueless causal paths that do not contribute to performance debugging.
• To improve the efficiency and effectiveness of request tracing in production data centers.
• To enable scalable, low-overhead performance debugging in distributed systems.

Proposed method

• The proposed algorithm analyzes request flow across service tiers to identify and filter out logs that do not contribute to causal path reconstruction.
• It leverages dependency and execution context analysis to distinguish between informative and redundant log entries.
• The method applies a pruning strategy based on causal path relevance, removing paths that do not provide diagnostic insight.
• It uses lightweight instrumentation to collect minimal yet sufficient metadata for path correlation and filtering.
• The algorithm operates in real time, enabling on-the-fly reduction of log volume during service execution.
• It integrates with existing tracing infrastructures without requiring architectural overhaul.

Experimental results

Research questions

• RQ1 How can log data volume be reduced in large-scale multi-tier services without losing essential debugging information?
• RQ2 What criteria can be used to identify and eliminate valueless causal paths in request tracing?
• RQ3 To what extent can redundant logs be filtered out while preserving end-to-end request traceability?
• RQ4 Can a lightweight, scalable algorithm effectively reduce log overhead in production data centers?
• RQ5 What is the impact of log reduction on the accuracy and completeness of performance debugging?

Key findings

• The proposed method filters out 84% of valueless causal paths in multi-tier service environments.
• The reduction in log data significantly decreases storage and processing overhead in large-scale deployments.
• The algorithm maintains high fidelity in request tracing by preserving only relevant diagnostic paths.
• The approach is effective in real-time production settings with minimal performance impact.
• The method demonstrates strong potential for integration into existing distributed tracing systems.
• The results indicate that a large portion of collected logs in multi-tier systems are non-contributing to debugging, making filtering highly effective.