[Paper Review] Case Studies of Causal Discovery from IT Monitoring Time Series

Ali Aït-Bachir, Karim ASSAAD+6 more|arXiv (Cornell University)|Jul 28, 2023

Data Quality and Management4 citations

TL;DR

This paper presents case studies applying causal discovery algorithms to real-world IT monitoring time series, demonstrating their potential for root cause analysis and proactive incident prediction despite challenges like misaligned data, missing values, and non-stationarity. The hybrid method NBCB-e achieves the highest F1-score (0.45) across datasets, indicating promise but also limited performance due to data complexity and algorithmic assumptions.

ABSTRACT

Information technology (IT) systems are vital for modern businesses, handling data storage, communication, and process automation. Monitoring these systems is crucial for their proper functioning and efficiency, as it allows collecting extensive observational time series data for analysis. The interest in causal discovery is growing in IT monitoring systems as knowing causal relations between different components of the IT system helps in reducing downtime, enhancing system performance and identifying root causes of anomalies and incidents. It also allows proactive prediction of future issues through historical data analysis. Despite its potential benefits, applying causal discovery algorithms on IT monitoring data poses challenges, due to the complexity of the data. For instance, IT monitoring data often contains misaligned time series, sleeping time series, timestamp errors and missing values. This paper presents case studies on applying causal discovery algorithms to different IT monitoring datasets, highlighting benefits and ongoing challenges.

Motivation & Objective

• To evaluate the effectiveness of causal discovery algorithms on real IT monitoring time series data.
• To identify challenges in applying causal discovery to complex, real-world IT monitoring data such as misaligned time series, missing values, and non-stationarity.
• To compare the performance of various causal discovery methods—including constraint-based, score-based, and hybrid approaches—across diverse IT monitoring datasets.
• To assess the impact of data preprocessing strategies and algorithmic assumptions (e.g., linearity, acyclicity) on causal discovery outcomes.
• To guide future research by identifying gaps in current methods, especially regarding non-linear relationships, mixed data types, and hidden confounders.

Proposed method

• The study applies multiple causal discovery algorithms, including PCMCI+, VLiNGAM, TiMINo, GCMVL, and hybrid methods like NBCB-e and CBNB-e, to real IT monitoring datasets.
• Hybrid methods combine constraint-based (PCMCI+ or PCGCE) with structure learning (VLiNGAM) to improve performance on complex time series.
• The full time causal graph is used as a theoretical foundation, though practical inference relies on windowed or summary causal graphs due to data limitations.
• Data preprocessing strategies are applied, including handling missing values and aligning time series, to improve algorithmic input quality.
• Performance is evaluated using F1-scores on causal edge detection, with results compared across datasets and preprocessing methods.
• The analysis considers assumptions like stationarity, consistency, and acyclicity, and evaluates their violation in real IT data.

Experimental results

Research questions

• RQ1 How do different causal discovery algorithms perform on real IT monitoring time series with complex data characteristics?
• RQ2 To what extent do data preprocessing strategies improve causal discovery performance on IT monitoring data?
• RQ3 Why do current causal discovery algorithms underperform on IT monitoring datasets, and what assumptions are most likely violated?
• RQ4 Can hybrid causal discovery methods outperform standalone approaches in real-world IT monitoring scenarios?
• RQ5 What role do data characteristics—such as non-linearity, mixed variable types, and hidden confounders—play in limiting algorithm effectiveness?

Key findings

• NBCB-e achieves the highest F1-score of 0.45 on the Antivirus 2 dataset, outperforming other methods across most datasets.
• VLiNGAM and TiMINo show poor performance on most datasets, particularly MoM 1 and MoM 2, indicating limitations in handling real-world IT monitoring data.
• Preprocessing Strategy 2 improves performance for all algorithms except GCMVL and TiMINo on the Antivirus dataset, suggesting data quality significantly affects results.
• The best-performing algorithm, NBCB-e, still achieves only moderate F1-scores, indicating that current causal discovery methods are far from optimal for real IT monitoring data.
• The study identifies violations of core assumptions—such as stationarity, linearity, and acyclicity—as major contributors to poor performance.
• The presence of mixed data types (continuous, ordinal, nominal) and hidden confounders further challenges existing methods, highlighting the need for more robust, flexible algorithms.