[Paper Review] Benchpress: A Scalable and Versatile Workflow for Benchmarking Structure Learning Algorithms

Felix L. Rios, Giusi Moffa+1 more|Zenodo (CERN European Organization for Nuclear Research)|Jul 8, 2021

Bayesian Modeling and Causal Inference4 citations

TL;DR

Benchpress is a scalable, modular, and reproducible workflow for benchmarking structure learning algorithms in probabilistic graphical models, using a JSON-configured interface to unify diverse algorithms, data generators, and evaluation metrics across multiple programming languages. It enables systematic, platform-independent comparisons with standardized outputs, visualizations, and support for MCMC diagnostics, significantly reducing the effort required for benchmarking while enhancing reproducibility and extensibility in causal discovery research.

ABSTRACT

Describing the relationship between the variables in a study domain and modelling the data generating mechanism is a fundamental problem in many empirical sciences. Probabilistic graphical models are one common approach to tackle the problem. Learning the graphical structure for such models is computationally challenging and a fervent area of current research with a plethora of algorithms being developed. To facilitate the benchmarking of different methods, we present a novel Snakemake workflow, called Benchpress for producing scalable, reproducible, and platform-independent benchmarks of structure learning algorithms for probabilistic graphical models. Benchpress is interfaced via a simple JSON-file, which makes it accessible for all users, while the code is designed in a fully modular fashion to enable researchers to contribute additional methodologies. Benchpress currently provides an interface to a large number of state-of-the-art algorithms from libraries such as BDgraph, BiDAG, bnlearn, causal-learn, gCastle, GOBNILP, pcalg, r.blip, scikit-learn, TETRAD, and trilearn as well as a variety of methods for data generating models and performance evaluation. Alongside user-defined models and randomly generated datasets, the workflow also includes a number of standard datasets and graphical models from the literature, which may be included in a benchmarking study. We demonstrate the applicability of this workflow for learning Bayesian networks in five typical data scenarios. The source code and documentation is publicly available from http://benchpressdocs.readthedocs.io.

Motivation & Objective

• To address the lack of standardized, reproducible, and scalable benchmarking workflows for structure learning algorithms in probabilistic graphical models.
• To unify diverse algorithms implemented in different languages and frameworks into a single, interoperable benchmarking environment.
• To reduce the burden of manual script development for comparative studies by providing a modular, extensible, and platform-independent workflow.
• To support comprehensive evaluation using multiple metrics, visualizations, and diagnostics, including MCMC convergence checks and graph property analysis.
• To promote reproducibility in causal discovery research by enabling systematic, transparent, and shareable benchmarking across diverse data scenarios.

Proposed method

• The workflow is configured via a simple, human-readable JSON file, enabling users to define algorithms, data sources, evaluation metrics, and visualization modules without coding.
• It integrates state-of-the-art structure learning algorithms from 11 major libraries, including bnlearn, pcalg, GOBNILP, and gCastle, enabling cross-comparison across diverse methodological families.
• The framework supports both synthetic data generation (with user-defined or literature-based graphical models) and real-world datasets, ensuring broad applicability.
• Evaluation is performed using standardized metrics such as F1, TP, FP, FN, and computational time, with results stored in structured CSV format for downstream analysis.
• Visualization modules generate boxplots, ROC curves, pair-wise plots, estimated vs. true graph comparisons, and MCMC diagnostics (e.g., trace plots, autocorrelation, heatmaps of edge probabilities).
• MCMC-based inference is supported with burn-in, thinning, and functional tracking (e.g., number of edges, score), enabling convergence assessment and posterior estimation.

Experimental results

Research questions

• RQ1 How can a unified, scalable, and reproducible benchmarking workflow be designed to compare diverse structure learning algorithms across different programming languages and frameworks?
• RQ2 What is the impact of varying data scenarios—such as low sample size, high dimensionality, or non-Gaussian distributions—on the performance of structure learning algorithms?
• RQ3 How do different evaluation metrics (e.g., F1, precision, recall) rank the same algorithm across different data settings, and can a multi-metric evaluation improve benchmarking robustness?
• RQ4 To what extent can MCMC-based inference methods be effectively monitored and visualized within a standardized benchmarking framework?
• RQ5 How can the integration of real datasets and semisynthetic data with known structures enhance the validity and generalizability of benchmarking results?

Key findings

• Benchpress enables end-to-end benchmarking of 14 state-of-the-art structure learning algorithms across five distinct data scenarios, significantly reducing setup and execution overhead.
• The workflow successfully integrates heterogeneous algorithms from 11 different libraries, including both score-based and constraint-based methods, into a single, consistent evaluation pipeline.
• MCMC-based methods, such as bidag_itsearch and gcastle_notears, were effectively monitored using built-in diagnostics, including trace plots, autocorrelation, and edge probability heatmaps.
• The framework produced standardized, machine-readable CSV outputs and rich visualizations (e.g., ROC curves, graph comparisons) that facilitate transparent and reproducible reporting.
• The inclusion of both synthetic and real-world datasets—such as those from the literature and standard benchmark repositories—enhances the generalizability and practical relevance of benchmark results.
• The modular design allows seamless extension with new algorithms, data generators, and evaluation metrics, ensuring long-term maintainability and community adoption.