[Paper Review] Frequentist and Bayesian Confidence Limits

G. Zech|arXiv (Cornell University)|Jun 5, 2001

Gaussian Processes and Bayesian Inference3 citations

TL;DR

This paper compares frequentist and Bayesian methods for constructing confidence limits, arguing against classical methods due to violations of the Likelihood Principle and issues with coherence and precision. It proposes error limits based solely on the likelihood function, aligning with standard practices in high-energy physics and rejecting both classical approaches and extreme Bayesian methods with arbitrary priors.

ABSTRACT

Frequentist (classical) and the Bayesian approaches to the construction of confidence limits are compared. Various examples which illustrate specific problems are presented. The Likelihood Principle and the Stopping Rule Paradox are discussed. The performance of the different methods is investigated relative to the properties coherence, precision, bias, universality, simplicity. A proposal on how to define error limits in various cases are derived from the comparison. They are based on the likelihood function only and follow in most cases the general practice in high energy physics. Classical methods are not recommended because they violate the Likelihood Principle, they can produce physically inconsistent results, suffer from lack of precision and generality. Also the extreme Bayesian approach with arbitrary choice of the prior probability density or priors deduced from scaling laws is rejected.

Motivation & Objective

• To evaluate the performance of frequentist and Bayesian methods in constructing confidence limits.
• To identify shortcomings in classical methods, such as violation of the Likelihood Principle and production of physically inconsistent results.
• To assess the impact of prior selection in Bayesian approaches, especially arbitrary or scaling-law-derived priors.
• To propose a unified, principled approach to defining error limits based exclusively on the likelihood function.
• To improve coherence, precision, and universality in statistical inference for scientific measurements.

Proposed method

• The paper compares frequentist and Bayesian approaches using illustrative examples that highlight methodological flaws.
• It applies the Likelihood Principle as a criterion to evaluate methodological consistency and reject methods that violate it.
• It investigates the stopping rule paradox to demonstrate inconsistencies in frequentist inference.
• It evaluates methods based on key statistical properties: coherence, precision, bias, universality, and simplicity.
• It derives a proposal for defining error limits using only the likelihood function, avoiding reliance on subjective or arbitrary priors.
• The approach is aligned with standard practices in high-energy physics, where likelihood-based intervals are commonly used.

Experimental results

Research questions

• RQ1 How do frequentist and Bayesian methods compare in constructing confidence limits under standard statistical criteria?
• RQ2 In what ways do classical methods violate the Likelihood Principle and produce physically inconsistent results?
• RQ3 Why are extreme Bayesian approaches with arbitrary or scaling-law-derived priors considered unsuitable for reliable inference?
• RQ4 Can a unified method for defining error limits be developed that ensures coherence, precision, and universality?
• RQ5 To what extent do likelihood-based intervals align with established practices in high-energy physics?

Key findings

• Classical methods are rejected due to violations of the Likelihood Principle and their tendency to produce physically inconsistent results.
• Frequentist methods often lack precision and generality, undermining their reliability in scientific inference.
• Extreme Bayesian approaches with arbitrary or scaling-law-derived priors are rejected due to their lack of objectivity and potential for bias.
• The proposed method, based solely on the likelihood function, ensures coherence and aligns with standard practices in high-energy physics.
• The likelihood-based approach outperforms both classical and extreme Bayesian methods in terms of precision, universality, and simplicity.
• The paper concludes that error limits should be defined using the likelihood function alone, avoiding reliance on subjective or arbitrary priors.