[Paper Review] On Statistical Non-Significance

Alberto Abadie|arXiv (Cornell University)|Mar 1, 2018

Forecasting Techniques and Applications8 references3 citations

TL;DR

This paper challenges the conventional view that statistical significance carries more scientific value than non-significance, demonstrating through a limited information Bayes framework that non-significant results often convey more information—especially in large samples where point nulls are rarely plausible. It argues that failure to reject a null hypothesis can sharply concentrate posterior belief around zero, while rejection may carry little information, advocating for greater visibility and discussion of non-significant findings in empirical research.

ABSTRACT

Significance tests are probably the most extended form of inference in empirical research, and significance is often interpreted as providing greater informational content than non-significance. In this article we show, however, that rejection of a point null often carries very little information, while failure to reject may be highly informative. This is particularly true in empirical contexts where data sets are large and where there are rarely reasons to put substantial prior probability on a point null. Our results challenge the usual practice of conferring point null rejections a higher level of scientific significance than non-rejections. In consequence, we advocate a visible reporting and discussion of non-significant results in empirical practice.

Motivation & Objective

• To challenge the widespread assumption that statistical significance is more informative than non-significance in empirical research.
• To demonstrate, using a limited information Bayes framework, that non-rejection of a point null can convey substantial information, especially in large samples.
• To show that point null rejection often carries minimal informational value when prior belief in the exact null is negligible.
• To advocate for the visible reporting and discussion of non-significant results in empirical practice, countering publication bias.
• To re-evaluate the role of significance testing by incorporating the sign of the estimate and its impact on posterior informativeness.

Proposed method

• Adopts a limited information Bayes approach, where agents update beliefs using only reported significance levels, not full data.
• Models the prior distribution of the parameter θ as Normal(μ, σ²), and the sampling distribution of the estimator ŷ as Normal(θ, 1/n).
• Derives the limited information posterior for θ conditional on significance (|√nŷ| > c) and non-significance (|√nŷ| ≤ c), using the cumulative distribution function Φ.
• Calculates the posterior ratio p(θ | data) / p(θ) to assess informativeness, particularly in the large-sample limit (n → ∞).
• Extends the analysis by conditioning on the sign of the estimate (e.g., √nŷ > c and ŷ > 0), showing how sign information alters posterior concentration.
• Uses asymptotic limits to analyze the behavior of posteriors under significance and non-significance, especially regarding concentration at θ = 0.

Experimental results

Research questions

• RQ1 Does statistical significance provide more information than non-significance in typical empirical settings with large samples?
• RQ2 How does the informativeness of a significance test depend on the prior distribution of the parameter of interest?
• RQ3 What is the impact of conditioning on the sign of the estimated coefficient in addition to significance?
• RQ4 How does the posterior distribution behave asymptotically under non-significance, especially regarding concentration at the null value?
• RQ5 Why is non-significance often underreported, and what are the implications for scientific inference and publication bias?

Key findings

• Non-significance can be highly informative, concentrating posterior probability mass sharply around the null value (θ = 0), especially in large samples.
• Under non-significance and positive sign, the posterior-to-prior ratio converges to infinity at θ = 0 and zero elsewhere, indicating extreme informativeness.
• Under significance with a positive sign, the posterior converges to the prior truncated at zero, limiting informativeness unless prior belief in θ > 0 is weak.
• The maximum informativeness of significance is bounded—the posterior ratio cannot exceed double the prior, even asymptotically, when μ ≥ 0.
• In large samples, the informational content of significance depends critically on prior belief in the sign of the parameter; when prior belief is strong, significance adds little new information.
• Failure to reject the null often carries more scientific weight than rejection, especially when the prior assigns little probability to the exact null value.