[Paper Review] Testing in the Presence of Nuisance Parameters: Some Comments on Tests Post-Model-Selection and Random Critical Values

Hannes Leeb, Benedikt M. Pötscher|Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)|Sep 20, 2012

Statistical Methods and Inference4 citations

TL;DR

This paper critiques the use of random critical values in hypothesis testing when nuisance parameters are present, demonstrating that such tests often exceed their nominal size, even asymptotically. It shows that while intuitive, procedures using estimated nuisance parameters as random critical values fail to maintain correct size, and instead advocates for conservative, supremum-based critical values to ensure valid inference.

ABSTRACT

We point out that the ideas underlying some test procedures recently proposed for testing post-model-selection (and for some other test problems) in the econometrics literature have been around for quite some time in the statistics literature. We also sharpen some of these results in the statistics literature. Furthermore, we show that some intuitively appealing testing procedures, that have found their way into the econometrics literature, lead to tests that do not have desirable size properties, not even asymptotically.

Motivation & Objective

• To identify and analyze the flaws in testing procedures that use random critical values derived from estimated nuisance parameters.
• To demonstrate that such tests can have size strictly greater than the nominal level, even asymptotically.
• To clarify that ideas underlying recent econometric proposals for post-model-selection testing are not novel but were previously established in the statistical literature.
• To provide sharper theoretical results on size control in the presence of nuisance parameters, particularly in non-regular estimation settings.
• To contrast the performance of random critical value tests with conservative, supremum-based alternatives that maintain correct size.

Proposed method

• Uses a framework where the test statistic's null distribution depends on an unknown nuisance parameter β, requiring critical values that account for this dependence.
• Proposes the use of a supremum-based critical value: $ c_{n, ext{sup}}(\delta) = \sup_{\beta \in B} c_{n,\beta}(\delta) $, ensuring the test has size ≤ δ.
• Analyzes the size properties of random critical values $ c_{n,\hat{\beta}_n}(\delta) $, where $ \hat{\beta}_n $ is an estimator of β.
• Employs a formal proof showing that if $ c_{n,\hat{\beta}_n}(\delta) < c_{n,\text{sup}}(\delta) $ with positive probability under the worst-case β, then the test size exceeds δ.
• Applies the result to a linear regression model with a known variance and correlated parameters, using asymptotic normal approximations to derive explicit expressions for critical values.
• Considers modified procedures with data-dependent thresholds $ \eta_n \to 0 $, showing that while asymptotic size control is approached, finite-sample size inflation persists unless carefully constructed.

Experimental results

Research questions

• RQ1 Do testing procedures that use random critical values based on estimated nuisance parameters maintain their nominal size in finite samples?
• RQ2 Why do some intuitively appealing tests using estimated nuisance parameters fail to control size, even asymptotically?
• RQ3 What is the theoretical justification for using supremum-based critical values in the presence of nuisance parameters?
• RQ4 How do the size properties of random critical value tests compare to those of conservative, fixed critical value tests?
• RQ5 Can data-dependent adjustments to random critical values restore correct size control in finite samples?

Key findings

• The test using random critical values $ c_{n,\hat{\beta}_n}(\delta) $ does not have level δ, as it can have size strictly greater than δ, even asymptotically.
• Condition (4) — that $ P_{n,\alpha_0,\beta_n^{\max}}(c_{n,\hat{\beta}_n}(\delta) < T_n(\alpha_0) \leq c_{n,\text{sup}}(\delta)) > 0 $ — is sufficient for size inflation, and this condition holds under regular conditions.
• The supremum-based critical value $ c_{n,\text{sup}}(\delta) $ ensures a valid level δ test, but at the cost of reduced power due to conservatism.
• Even with $ \eta_n \to 0 $, the test based on $ \hat{c}_{n,\eta_n,\min}(\delta) $ continues to exceed the nominal size for all finite n, despite approaching the conservative test asymptotically.
• The test based on $ c_{n,\eta_n,\text{Loh}}(\delta) $, which uses a data-dependent threshold, maintains correct size for each n and approaches the conservative test as $ \eta_n \to 0 $, making it preferable to alternatives.
• The result that random critical values can lead to invalid tests was previously established in statistics literature (e.g., Loh, 1985), and the paper sharpens these findings in the context of post-model-selection inference.