[Paper Review] Flow-Bench: A Dataset for Computational Workflow Anomaly Detection

George Papadimitriou, Hongwei Jin+6 more|arXiv (Cornell University)|Jun 16, 2023

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This paper introduces Flow-Bench, a novel benchmark dataset for computational workflow anomaly detection, created by injecting synthetic anomalies into real workflows executed on distributed HPC systems. It evaluates unsupervised ML techniques on tabular and graph-structured workflow data, revealing that while some methods achieve moderate performance (e.g., F1 up to 0.663), many fail due to memory issues or poor precision, highlighting the need for domain-specific anomaly detection approaches in scientific workflows.

ABSTRACT

A computational workflow, also known as workflow, consists of tasks that must be executed in a specific order to attain a specific goal. Often, in fields such as biology, chemistry, physics, and data science, among others, these workflows are complex and are executed in large-scale, distributed, and heterogeneous computing environments prone to failures and performance degradation. Therefore, anomaly detection for workflows is an important paradigm that aims to identify unexpected behavior or errors in workflow execution. This crucial task to improve the reliability of workflow executions can be further assisted by machine learning-based techniques. However, such application is limited, in large part, due to the lack of open datasets and benchmarking. To address this gap, we make the following contributions in this paper: (1) we systematically inject anomalies and collect raw execution logs from workflows executing on distributed infrastructures; (2) we summarize the statistics of new datasets, and provide insightful analyses; (3) we convert workflows into tabular, graph and text data, and benchmark with supervised and unsupervised anomaly detection techniques correspondingly. The presented dataset and benchmarks allow examining the effectiveness and efficiency of scientific computational workflows and identifying potential research opportunities for improvement and generalization. The dataset and benchmark code are publicly available \url{https://poseidon-workflows.github.io/FlowBench/} under the MIT License.

Motivation & Objective

• To address the lack of open, standardized benchmark datasets for evaluating anomaly detection in computational workflows.
• To generate realistic, diverse workflow anomaly data by injecting synthetic faults into real scientific and data science workflows.
• To evaluate the performance of state-of-the-art unsupervised anomaly detection techniques on both tabular and graph-structured representations of workflows.
• To identify limitations and scalability issues in existing anomaly detection methods when applied to large-scale, complex workflows.
• To provide a public, MIT-licensed dataset and codebase to enable reproducible research and future development in workflow anomaly detection.

Proposed method

• Systematically inject anomalies—such as task delays, failures, and resource overloads—into real-world scientific and data science workflows running on distributed HPC infrastructures.
• Collect raw execution logs from workflows executed on heterogeneous, large-scale computing environments, including systems at USC, ANL, RENCI, and ORNL.
• Transform workflow data into both tabular and graph-structured formats, modeling dependencies as directed acyclic graphs (DAGs) for graph-based anomaly detection.
• Benchmark 15 state-of-the-art unsupervised anomaly detection methods, including PyOD and PyGOD techniques, on the resulting datasets.
• Evaluate methods using standard metrics: ROC-AUC, F1-score, top-k precision, and memory efficiency (e.g., out-of-memory errors).
• Provide comprehensive statistical summaries of the new dataset and existing open datasets to support comparative analysis and method development.

Experimental results

Research questions

• RQ1 How effective are existing unsupervised anomaly detection techniques when applied to real computational workflows with synthetic anomalies?
• RQ2 What are the scalability and memory efficiency limitations of current anomaly detection methods on large-scale workflow data?
• RQ3 How do performance metrics like F1-score and ROC-AUC vary across different workflow types (e.g., simulation, data analysis, ML)?
• RQ4 To what extent do graph-based versus tabular-based representations improve anomaly detection performance?
• RQ5 Can the proposed benchmark dataset support the development of more robust and generalizable anomaly detection models for scientific workflows?

Key findings

• Many standard anomaly detection methods, including LOF and OCSVM, suffer from out-of-memory (OOM) errors when processing even moderately sized workflow data, indicating scalability issues.
• The best-performing method, KNN, achieved an F1-score of 0.663 on the data science workflow, but other methods like LOF and INNE achieved F1-scores below 0.10, indicating poor detection performance.
• Graph-based methods such as GNNs showed promise but were outperformed by some tabular methods, suggesting that graph structure alone may not be sufficient for effective anomaly detection.
• The LMDD method failed due to time-to-live errors (TLE), highlighting instability in certain algorithms under real-world computational constraints.
• The dataset reveals significant class imbalance, with normal instances vastly outnumbering anomalies, which challenges the generalization of unsupervised models.
• Raw log data contains rich contextual information that could be leveraged by NLP-based models, suggesting a path for future research beyond graph and tabular representations.