[Paper Review] Consistent, multidimensional differential histogramming and summary statistics with YODA 2

A. G. Buckley, Corpe, Louie+6 more|arXiv (Cornell University)|Dec 22, 2023

Computational Physics and Python Applications4 citations

TL;DR

This paper presents YODA 2, a modern, type-safe C++ library for consistent, multidimensional differential histogramming and summary statistics, designed for high-performance computing in particle physics. It enables efficient, numerically robust aggregation of large datasets through separation of binning logic from statistical content, with first-class support for Python plotting via text-based script generation and extensible data formats including HDF5 and MPI serialization.

ABSTRACT

Histogramming is often taken for granted, but the power and compactness of partially aggregated, multidimensional summary statistics, and their fundamental connection to differential and integral calculus make them formidable statistical objects, especially when very large data volumes are involved. But expressing these concepts robustly and efficiently in high-dimensional parameter spaces and for large data samples is a highly non-trivial challenge -- doubly so if the resulting library is to remain usable by scientists as opposed to software engineers. In this paper we summarise the core principles required for consistent generalised histogramming, and use them to motivate the design principles and implementation mechanics of the re-engineered YODA histogramming library, a key component of physics data-model comparison and statistical interpretation in collider physics.

Motivation & Objective

• Address the limitations of existing histogramming libraries in handling high-dimensional, large-scale data with consistent statistical semantics.
• Design a library that maintains mathematical rigor in differential histogramming while remaining accessible to physicists and scientists, not just software engineers.
• Enable efficient, scalable, and parallelizable histogram aggregation across distributed and high-performance computing environments.
• Support flexible data exchange, persistency, and visualization through standardized, human-readable, and extensible formats.
• Provide a unified, extensible framework for statistical analysis that supports future extensions like systematic uncertainties and unbinned data types.

Proposed method

• Decouple bin partitioning from statistical content using a two-layer design: live (mutable) and inert (immutable) data types.
• Use template metaprogramming in C++ to enforce type safety and enable compile-time optimization of statistical operations.
• Implement consistent, numerically stable computation of weighted and unweighted moments (mean, variance) using Bessel-corrected estimators.
• Support multidimensional histogramming via recursive binning and projection of second-order cumulants with exact conversion between dimensions.
• Generate human-readable, executable Python plotting scripts from histogram data, enabling reproducible and customizable visualizations via Matplotlib.
• Provide over-the-wire serialization for MPI and support for persistent storage in a custom text-based format (with HDF5 in development) for portability and performance.

Figure 5: Example 1D and 2D plot outputs from the Y ODA 2 plotting interface, illustrating a mix of uniform and irregular bin sizes, and automatic use of ratio plotting for 1D data/prediction comparisons.

Experimental results

Research questions

• RQ1 How can histogramming be made both mathematically consistent and computationally efficient in high-dimensional parameter spaces?
• RQ2 What design principles enable a histogramming library to be both performant and user-friendly for physicists without requiring deep software engineering expertise?
• RQ3 How can statistical moments be aggregated robustly across large, distributed datasets while preserving numerical accuracy?
• RQ4 What mechanisms allow for efficient, lossless conversion between different projections of multidimensional histograms?
• RQ5 How can histogram data be persistently stored and shared across platforms and computational environments with minimal coupling?

Key findings

• YODA 2 successfully decouples binning logic from statistical content, enabling efficient, type-safe, and numerically stable histogram aggregation in C++ with compile-time optimizations.
• The library supports consistent, multidimensional differential histogramming with exact conversion between projections of second-order statistical cumulants, ensuring mathematical fidelity.
• Live and inert data types are strictly separated with one-way conversion, reducing runtime errors and enabling efficient memory management in high-throughput workloads.
• The Python plotting interface generates transparent, executable scripts that interface directly with Matplotlib, enabling reproducible, customizable, and collaborative visualization workflows.
• The library is production-ready and already in use with the Rivet MC analysis framework in high-energy physics, demonstrating scalability and robustness in real-world HEP applications.
• Future support for HDF5 persistency and multi-weighted systematic uncertainties is planned, with extensibility for unbinned data types and continued evolution via modern C++ template techniques.