[Paper Review] A method for statistical comparison of histograms

Sergey Bityukov, Nikolai Krasnikov+2 more|arXiv (Cornell University)|Feb 11, 2013

Advanced Data Processing Techniques3 references5 citations

TL;DR

This paper proposes a statistical method for comparing histograms by computing the significance of deviation between bin-by-bin values, using normalized significance and RMS of these significances as test statistics. The approach enables robust estimation of the statistical difference between two histograms, with superior distinguishability performance compared to traditional chi-squared tests, especially when total event counts differ or distributions vary across bins.

ABSTRACT

We propose an approach for testing the hypothesis that two realizations of the random variables in the form of histograms are taken from the same statistical population (i.e. that two histograms are drawn from the same distribution). The approach is based on the notion "significance of deviation". Our approach allows also to estimate the statistical difference between two histograms.

Motivation & Objective

• To develop a statistical method for testing whether two histograms are drawn from the same underlying distribution.
• To estimate the statistical difference between two histograms beyond simple goodness-of-fit tests.
• To provide a more sensitive and interpretable measure of distinguishability between histograms, especially when total event counts differ.
• To enable practical application in experimental monitoring, such as in high-energy physics experiments.

Proposed method

• The method computes the bin-wise significance of deviation between two histogram bins using the formula $\hat{S}_i = \frac{\hat{n}_{i1} - \hat{n}_{i2}}{\sqrt{\hat{\sigma}^2_{n_{i1}} + \hat{\sigma}^2_{n_{i2}}}}$, where $\hat{n}_{ik}$ is the observed count and variance is approximated as $\hat{n}_{ik}$.
• The root mean square (RMS) of the bin-wise significances is used as a global distance measure between histograms.
• For cases with different total event counts, a normalized significance is applied: $\hat{S}_i(K) = \frac{\hat{n}_{i1} - K\hat{n}_{i2}}{\sqrt{\hat{\sigma}^2_{n_{i1}} + K^2\hat{\sigma}^2_{n_{i2}}}}$, with $K = N_1/N_2$.
• The distribution of normalized significances is shown to approximate a standard normal distribution under the null hypothesis.
• The method uses the first moment ($\bar{S}$) and second central moment (RMS) of the significance distribution to assess distinguishability.
• The probability of correct decision ($1 - \kappa$) is used as a measure of distinguishability, derived from Type I and Type II error rates.

Experimental results

Research questions

• RQ1 Can the significance of deviation between two histogram bins be used as a reliable test statistic for comparing entire histograms?
• RQ2 How does the distribution of bin-wise significances behave under the null hypothesis that two histograms are from the same population?
• RQ3 Does the RMS of significances provide a more sensitive measure of histogram difference than the standard $\chi^2$ test?
• RQ4 How does normalization for differing total event counts affect the reliability and interpretability of the significance-based comparison?
• RQ5 To what extent can the joint distribution of $\bar{S}$ and RMS improve the distinguishability of histograms compared to single-statistic methods?

Key findings

• The distribution of bin-wise significances $\hat{S}_i$ closely follows a standard normal distribution when histograms are from the same statistical population.
• The RMS of significances provides a robust global measure of the statistical difference between two histograms.
• For histograms with different total event counts, the normalized significance $\hat{S}_i(K)$ yields a distribution close to standard normal, ensuring valid inference.
• The method based on $\bar{S}$ and RMS achieves a 93.88% probability of correct decision in distinguishing histograms, outperforming the $\chi^2$ method (87.26%).
• The approach enables reliable monitoring of experimental equipment by detecting subtle deviations in histogram shapes even when total event counts differ.
• The use of pseudo-populations from Monte Carlo simulations allows estimation of distinguishability for any pair of histograms under study.