[Paper Review] On Stepwise Control of Directional Errors under Independence and Some Dependence

Wenge Guo, Joseph P. Romano|arXiv (Cornell University)|Feb 9, 2016

Statistical Methods in Clinical Trials22 references4 citations

TL;DR

This paper develops new stepwise multiple testing procedures that jointly control type 1 and directional (type 3) errors under independence and various dependence structures. By formulating the problem to simultaneously control mixed directional familywise error rate (mdFWER) and mixed directional false discovery rate (mdFDR), the authors propose procedures that achieve strong control at level α, extending prior work under restrictive distributional assumptions.

ABSTRACT

In this paper, the problem of error control of stepwise multiple testing procedures is considered. For two-sided hypotheses, control of both type 1 and type 3 (or directional) errors is required, and thus mixed directional familywise error rate control and mixed directional false discovery rate control are each considered by incorporating both types of errors in the error rate. Mixed directional familywise error rate control of stepwise methods in multiple testing has proven to be a challenging problem, as demonstrated in Shaffer (1980). By an appropriate formulation of the problem, some new stepwise procedures are developed that control type 1 and directional errors under independence and various dependencies.

Motivation & Objective

• To address the challenge of simultaneously controlling both type 1 and directional (type 3) errors in multiple two-sided hypothesis testing.
• To extend existing methods—previously limited to independent test statistics and specific distributions—by developing procedures that maintain error control under general dependence structures.
• To provide a unified framework for stepwise procedures that control the mixed directional familywise error rate (mdFWER) and mixed directional false discovery rate (mdFDR).
• To overcome limitations of prior approaches, such as the directional Holm procedure, which fails under non-normal or heavy-tailed distributions like Cauchy.
• To establish theoretical guarantees for mdFWER and mdFDR control using novel proof techniques under weak dependence assumptions.

Proposed method

• Formulates the multiple testing problem as controlling the joint error rate involving both type 1 and directional errors, defining mdFWER = Pr(U ≥ 1) and mdFDR = E(U / max(R,1)).
• Develops stepdown and stepup procedures using critical constants derived from α and sample size, with adjustments for dependence via conditional independence and TP3 density assumptions.
• Applies a novel proof strategy based on conditioning on the set of rejected false null hypotheses and leveraging symmetry in p-values under the null.
• Uses the identity P_{n+i} = 1 - P_i for paired one-sided hypotheses to simplify expectation calculations in FDR control proofs.
• Employs indicator functions and order statistics of p-values to bound the probability of false rejections and directional errors.
• Establishes control via recursive bounding of rejection probabilities under the null, using the structure of stepwise procedures and conditional independence.

Experimental results

Research questions

• RQ1 Can stepwise multiple testing procedures be constructed to jointly control both type 1 and directional errors under general dependence structures?
• RQ2 Does the directional Holm procedure maintain mdFWER control under non-normal or heavy-tailed distributions, such as Cauchy?
• RQ3 Can the mdFWER and mdFDR be controlled simultaneously using stepwise procedures under independence and positive regression dependency?
• RQ4 How do the proposed procedures compare in power to existing methods like the directional Hochberg or modified Bonferroni procedures?
• RQ5 What are the theoretical conditions under which stepwise procedures achieve strong control of mdFWER and mdFDR without requiring strict distributional assumptions?

Key findings

• The proposed stepwise procedures control the mixed directional familywise error rate (mdFWER) at level α under independence and certain dependence structures, including positive regression dependency.
• The authors establish that the directional Holm procedure maintains strong control of mdFWER under independence and mild distributional assumptions, resolving a long-standing open problem.
• For the mdFDR, the proposed stepup procedure ensures control at level α under independence and symmetric dependence, with the proof relying on the identity P_{n+i} = 1 - P_i.
• The paper provides a counterexample showing that the directional Holm procedure fails to control mdFWER under Cauchy-distributed test statistics, even under independence.
• The proof technique extends to a broad class of stepwise and closed testing procedures, including those with TP3 densities, offering a general framework for error control.
• The results demonstrate that improved stepdown procedures can significantly outperform single-step methods like the modified Bonferroni procedure, even when the latter controls FWER under arbitrary dependence.