[Paper Review] Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal's Fail-Safe Number

Konstantinos C. Fragkos, Michail Tsagris+1 more|Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)|Sep 4, 2015

Forecasting Techniques and Applications3 citations

TL;DR

This paper develops confidence intervals for Rosenthal's fail-safe number to assess publication bias in meta-analyses, using both normal approximation and nonparametric bootstrap methods. It derives distinct variance estimators for fixed and random numbers of studies, with the half-normal distribution estimator showing the best coverage probability in simulations.

ABSTRACT

The purpose of the present paper is to assess the efficacy of confidence intervals for Rosenthal's fail-safe number. Although Rosenthal's estimator is highly used by researchers, its statistical properties are largely unexplored. First of all, we developed statistical theory which allowed us to produce confidence intervals for Rosenthal's fail-safe number.This was produced by discerning whether the number of studies analysed in a meta-analysis is fixed or random. Each case produces different variance estimators. For a given number of studies and a given distribution, we provided five variance estimators. Confidence intervals are examined with a normal approximation and a nonparametric bootstrap. The accuracy of the different confidence interval estimates was then tested by methods of simulation under different distributional assumptions. The half normal distribution variance estimator has the best probability coverage. Finally, we provide a table of lower confidence intervals for Rosenthal's estimator.

Motivation & Objective

• To evaluate the statistical properties of Rosenthal's fail-safe number, which is widely used but poorly understood.
• To develop confidence intervals for the fail-safe number under fixed and random study count assumptions.
• To compare multiple variance estimators for robustness and accuracy in coverage probability.
• To assess the performance of normal approximation and bootstrap methods in constructing reliable intervals.
• To provide a practical table of lower confidence bounds for researchers using the fail-safe number.

Proposed method

• Derives theoretical variance estimators for Rosenthal's fail-safe number under fixed and random study count assumptions.
• Proposes five distinct variance estimators for a given number of studies and distributional assumptions.
• Applies normal approximation and nonparametric bootstrap techniques to construct confidence intervals.
• Employs simulation studies under various distributional assumptions to evaluate interval accuracy.
• Uses probability coverage as the primary metric to compare the performance of different interval estimation methods.
• Selects the half-normal variance estimator as optimal based on empirical coverage rates.

Experimental results

Research questions

• RQ1 How do confidence intervals for Rosenthal's fail-safe number perform under different distributional assumptions?
• RQ2 What is the impact of treating the number of studies as fixed versus random on variance estimation?
• RQ3 Which variance estimator yields the most accurate confidence interval coverage for the fail-safe number?
• RQ4 How do normal approximation and bootstrap methods compare in constructing reliable intervals?
• RQ5 What is the empirical performance of different interval estimation techniques in finite-sample settings?

Key findings

• The half-normal distribution variance estimator demonstrated the best probability coverage across simulation scenarios.
• Confidence intervals based on the half-normal estimator maintained coverage close to the nominal level, outperforming other estimators.
• The normal approximation method showed acceptable performance but was less reliable than the bootstrap under non-normal conditions.
• The bootstrap method provided more robust interval estimates, especially when distributional assumptions were violated.
• The study confirms that the fail-safe number's sampling distribution is sensitive to the choice of variance estimator.
• A table of lower confidence bounds for the fail-safe number is provided for practical use in meta-analytic research.