[Paper Review] Comparing Bayes factors and hierarchical inference for testing general relativity with gravitational waves

M. Isi, Will M. Farr+1 more|arXiv (Cornell University)|Apr 22, 2022

Pulsars and Gravitational Waves Research53 references25 citations

TL;DR

This paper demonstrates that combining Bayes factors from multiple gravitational wave events via naive multiplication can lead to misleading conclusions due to strong sensitivity to non-informative prior choices, especially in regions of parameter space with no likelihood support. In contrast, hierarchical inference—by modeling the population distribution of deviations from general relativity—provides a more robust and reliable method that converges to the correct conclusion regardless of prior width, avoiding the pitfalls of multiplicative Bayes factors.

ABSTRACT

In the context of testing general relativity with gravitational waves, constraints obtained with multiple events are typically combined either through a hierarchical formalism or though a combined multiplicative Bayes factor. We show that the well-known dependence of Bayes factors on the analysis priors in regions of the parameter space without likelihood support can lead to strong confidence in favor of incorrect conclusions when one employs the multiplicative Bayes factor. Bayes factors $\mathcal{O}(1)$ are ambivalent as they depend sensitively on the analysis priors, which are rarely set in a principled way; additionally, combined Bayes factors $>\mathcal{O}(10^3)$ can be obtained in favor of the incorrect conclusion depending on the analysis priors when many $\mathcal{O}(1)$ Bayes factors are multiplied, and specifically when the priors are much wider than the underlying population. The hierarchical analysis that instead infers the ensemble distribution of the individual beyond-general-relativity constraints does not suffer from this problem, and generically converges to favor the correct conclusion. Rather than a naive multiplication, a more reliable Bayes factor can be computed from the hierarchical analysis. We present a number of toy models showing that the practice of multiplying Bayes Factors can lead to incorrect conclusions.

Motivation & Objective

• To investigate the reliability of combining multiple gravitational wave event constraints using multiplicative Bayes factors.
• To identify the critical flaw in multiplicative Bayes factors: their extreme sensitivity to non-informative priors in regions of no likelihood support.
• To demonstrate that hierarchical inference, which learns the population distribution of deviations from general relativity, avoids this issue.
• To show that hierarchical modeling provides a more principled and robust alternative for multi-event testing of general relativity.
• To provide concrete toy models illustrating how multiplicative Bayes factors can incorrectly favor the wrong hypothesis due to prior choice.

Proposed method

• Uses toy models with known ground-truth distributions to simulate multiple gravitational wave detections.
• Compares multiplicative Bayes factors (BFs) from individual events to a hierarchical model that infers the population distribution of beyond-GR parameters.
• Applies a Gaussian moment expansion to model the population distribution of deviations, with mean μ and standard deviation σpop as hyperparameters.
• Employs hierarchical Bayesian inference to estimate μ and σpop from the data, treating the population distribution as a prior that adapts to observations.
• Analyzes the scaling behavior of multiplicative BFs with increasing numbers of events, showing their dependence on fixed, wide priors.
• Demonstrates that hierarchical posteriors converge to the correct conclusion even when priors are uninformative, unlike multiplicative BFs which can be arbitrarily biased.

Experimental results

Research questions

• RQ1 Can naive multiplication of Bayes factors from multiple gravitational wave events lead to incorrect conclusions due to prior sensitivity?
• RQ2 How does the performance of multiplicative Bayes factors scale with the number of events when priors are uninformative or widely specified?
• RQ3 Does hierarchical inference, which models the population distribution of deviations from general relativity, avoid the prior-dependent biases seen in multiplicative Bayes factors?
• RQ4 Can hierarchical modeling reliably recover the true underlying distribution of beyond-GR parameters even when individual event constraints are ambiguous?
• RQ5 What are the quantitative differences in inference reliability between multiplicative Bayes factors and hierarchical models in controlled toy scenarios?

Key findings

• Multiplicative Bayes factors can yield strong support for the incorrect hypothesis (e.g., O(10^3) in favor of a false deviation) when priors are much wider than the true population, even with many O(1) individual BFs.
• Bayes factors of order O(1) are inherently ambiguous and highly sensitive to prior specification, especially in regions of parameter space with no likelihood support.
• Hierarchical inference consistently converges to the correct conclusion by learning the population distribution from data, avoiding the prior-induced biases of multiplicative BFs.
• The hierarchical approach provides a more reliable and principled way to combine evidence across multiple events, as it does not assume identical or independent deviations across events.
• In all tested toy models, multiplicative BFs produced misleading results when priors were uninformative, while hierarchical inference correctly identified the true underlying model.
• The study shows that the choice of prior has a dominant influence on multiplicative BFs, whereas hierarchical models are robust to prior assumptions due to data-adaptive hyperpriors.