[Paper Review] Confidence intervals with control of the sign error in low power settings

Art B. Owen|arXiv (Cornell University)|Oct 31, 2016

Statistical Methods and Inference3 references3 citations

TL;DR

This paper proposes a method to control sign errors in low-power hypothesis tests by adjusting the significance level for confidence in the sign of a parameter estimate. By rejecting the null at a stricter level α₂ = 2α₁α_S, where α_S bounds the sign error probability, the method ensures that sign inferences are reliable even when power is low, offering a simple post-selection inference framework for parameter sign estimation.

ABSTRACT

When hypothesis tests of $H_0 heta=0$ have low power, it is possible that their rejection can frequently be accompanied by an estimate $\hat heta$ that has the wrong sign and significantly exaggerated magnitude. Such sign errors are less likely when the confidence interval for $ heta$ is well separated from 0, as measured in units of the confidence interval's width. Sign errors can be controlled by declaring confidence in the sign of $ heta$ only when $H_0$ is also rejected at a smaller level $\alpha_2=2\alpha_1\alpha_S$, where $\alpha_S\le 1/2$ is a user specified upper bound on the probability of a sign error given that $H_0$ has been rejected. This procedure is a very simple form of inference after model selection: the selected model is that $ heta e0$ and the second inference is then on sign($ heta$).

Motivation & Objective

• Address the risk of sign errors—where a parameter estimate has the wrong sign—when statistical power is low.
• Provide a method to control the probability of such sign errors given that the null hypothesis is rejected.
• Develop a practical, post-model-selection inference procedure for determining the sign of a parameter with controlled error rates.
• Offer a simple adjustment to standard confidence intervals to improve reliability in low-power settings.
• Ensure that confidence in the sign of a parameter is only declared when the evidence is sufficiently strong to minimize sign error risk.

Proposed method

• Propose a modified significance level α₂ = 2α₁α_S, where α₁ is the original test level and α_S is the user-specified upper bound on sign error probability.
• Declare confidence in the sign of the parameter θ only when the null hypothesis H₀: θ = 0 is rejected at the stricter level α₂.
• Use the width of the confidence interval as a measure of separation from zero to assess reliability of sign inference.
• Frame the procedure as a two-stage inference: first test H₀, then infer the sign only if H₀ is rejected at α₂.
• Integrate the sign error control into a simple, interpretable framework suitable for post-model-selection inference.
• Ensure that the probability of a sign error, given rejection of H₀, is bounded by α_S through the choice of α₂.

Experimental results

Research questions

• RQ1 How can sign errors in low-power hypothesis tests be controlled when the estimated parameter has the wrong sign despite rejecting the null?
• RQ2 What significance level adjustment is required to bound the probability of a sign error given that the null hypothesis is rejected?
• RQ3 Can a simple, post-selection inference procedure reliably determine the sign of a parameter in low-power settings?
• RQ4 How does the width of the confidence interval relate to the reliability of sign inference in low-power scenarios?
• RQ5 What is the relationship between the original significance level α₁ and the adjusted level α₂ needed to control sign error rates?

Key findings

• The method controls the sign error probability at or below the user-specified bound α_S, given that the null hypothesis is rejected.
• By setting α₂ = 2α₁α_S, the procedure ensures that sign inferences are made only when the evidence is strong enough to minimize the chance of a wrong sign conclusion.
• The approach is robust in low-power settings where standard confidence intervals may be misleading due to high sign error rates.
• The confidence interval's width relative to zero serves as a useful indicator of the reliability of sign inference, with greater separation reducing sign error risk.
• The method provides a principled, simple way to perform inference on the sign of a parameter after model selection, even when power is low.
• The procedure is computationally straightforward and does not require complex resampling or simulation, making it practical for routine use.