[Paper Review] Statistical Issues in Neutrino Physics Analyses

L. Lyons|arXiv (Cornell University)|Jul 13, 2016

Particle physics theoretical and experimental studies5 references3 citations

TL;DR

This paper addresses critical statistical challenges in neutrino physics, including parameter estimation, hypothesis testing, and coverage issues, advocating for rigorous methods like likelihood-based inference and blind analysis to ensure reliable results. It emphasizes avoiding common pitfalls such as biased parameter combination and undercoverage in confidence intervals.

ABSTRACT

Various statistical issues relevant to searches for new physics or to parameter determination in analyses of data in neutrino experiments are briefly discussed.

Motivation & Objective

• To address statistical pitfalls in neutrino experiments that can lead to incorrect conclusions or overconfident results.
• To promote robust statistical practices—such as blind analysis and proper coverage calibration—to ensure reliability in parameter estimation and hypothesis testing.
• To highlight the limitations of standard methods like BLUE and chi-squared when uncertainties are estimated from data, and to advocate for likelihood-based or iterative approaches.
• To guide researchers in selecting appropriate statistical tools for new physics searches and parameter determination in high-precision neutrino experiments.

Proposed method

• Uses likelihood-based inference with the Δ(lnL) = 1/2 rule for confidence interval construction, avoiding undercoverage issues common in Poisson parameter estimation.
• Advocates for the Feldman-Cousins method and Neyman construction to achieve proper coverage, especially in low-statistics scenarios.
• Promotes blind analysis techniques to prevent subconscious bias in data interpretation, particularly in searches for new physics.
• Recommends combining data at the event level rather than combining published results to preserve correlations and improve precision.
• Applies Wilks’ Theorem for nested hypothesis testing, using the difference in log-likelihoods (ΔS) as a test statistic distributed as χ² under the null hypothesis.
• Encourages the use of multivariate techniques and probability density functions from Monte Carlo simulations to distinguish between competing hypotheses (e.g., normal vs. inverted mass hierarchy).

Experimental results

Research questions

• RQ1 How can statistical methods be improved to avoid undercoverage in confidence intervals for Poisson-distributed neutrino event counts?
• RQ2 What are the risks of combining results using BLUE when uncertainties are estimated from data, and how can this be mitigated?
• RQ3 In what ways can blind analysis prevent bias in neutrino experiments, especially during new physics searches?
• RQ4 How should systematic uncertainties and correlations be properly accounted for when combining measurements of neutrino parameters?
• RQ5 When is Wilks’ Theorem applicable for hypothesis testing in neutrino physics, and what are the consequences of violating its assumptions?

Key findings

• The standard Δ(lnL) = 1/2 rule for confidence intervals in Poisson parameter estimation leads to non-constant coverage, with regions where the actual coverage falls significantly below the nominal confidence level.
• Combining results using BLUE can produce counterintuitive outcomes—such as a combined estimate of 2±1 from 100±10 and 1±1 counts—when estimated uncertainties are used as weights, due to statistical fluctuations.
• Iterative BLUE or full likelihood methods are superior to standard BLUE in avoiding such artifacts, especially when uncertainties are poorly estimated.
• When two measurements of a parameter are highly correlated (ρ > σ₁/σ₂), the combined estimate may lie outside the range of the individual measurements, which is statistically reasonable but sensitive to error in ρ or σ.
• Combining data from multiple experiments at the event level, rather than combining published results, can yield significantly smaller combined uncertainties, especially when correlations between measurements differ.
• Wilks’ Theorem provides a valid framework for hypothesis testing between nested models (e.g., normal vs. inverted mass hierarchy), but only if the assumptions—nested hypotheses, well-defined parameters, and asymptotic data—are satisfied; otherwise, simulation is required to determine the test statistic distribution.