[Paper Review] A Unified Treatment of Multiple Testing with Prior Knowledge

Aaditya Ramdas, Rina Foygel Barber+2 more|arXiv (Cornell University)|Mar 18, 2017

Statistical Methods in Clinical Trials7 citations

TL;DR

This paper introduces p-filter, a unified algorithmic framework for multiple hypothesis testing that simultaneously incorporates four types of prior knowledge: non-uniform prior weights, varying false discovery penalties, multiple overlapping group partitions, and dependence structures. By integrating these elements, p-filter enhances statistical power and precision, recovering known methods as special cases while enabling more adaptive and informed FDR control and global null testing.

ABSTRACT

There is a significant literature on methods for incorporating knowledge into multiple testing procedures so as to improve their power and precision. Some common forms of prior knowledge include (a) beliefs about which hypotheses are null, modeled by non-uniform prior weights; (b) differing importances of hypotheses, modeled by differing penalties for false discoveries; (c) multiple arbitrary partitions of the hypotheses into (possibly overlapping) groups; and (d) knowledge of independence, positive or arbitrary dependence between hypotheses or groups, suggesting the use of more aggressive or conservative procedures. We present a unified algorithmic framework called p-filter for global null testing and false discovery rate (FDR) control that allows the scientist to incorporate all four types of prior knowledge (a)-(d) simultaneously, recovering a variety of known algorithms as special cases.

Motivation & Objective

• To address the challenge of improving statistical power and precision in multiple hypothesis testing by leveraging diverse forms of prior knowledge.
• To unify existing methods that incorporate prior knowledge—such as prior weights, false discovery penalties, group structures, and dependence assumptions—into a single coherent framework.
• To develop a flexible algorithm that allows scientists to simultaneously use multiple types of prior knowledge in a principled way.
• To enable more accurate and powerful false discovery rate (FDR) control and global null testing by integrating heterogeneous prior information.

Proposed method

• p-filter employs a unified algorithmic framework that combines p-values with prior knowledge through a flexible optimization procedure.
• It models prior beliefs about null hypotheses using non-uniform prior weights to prioritize testing of more plausible alternatives.
• It incorporates differing penalties for false discoveries by assigning distinct error costs to hypotheses based on their importance.
• It supports multiple arbitrary partitions of hypotheses into groups, even overlapping ones, to reflect domain-specific clustering or structural knowledge.
• It accounts for dependence structures—such as independence, positive, or arbitrary dependence—between hypotheses or groups to adjust error control strategies accordingly.
• The framework uses a unified objective function that balances FDR control and power, integrating all four types of prior knowledge into a single optimization.

Experimental results

Research questions

• RQ1 How can multiple forms of prior knowledge be systematically integrated into multiple testing procedures to improve power and precision?
• RQ2 To what extent can a single framework unify existing methods that incorporate prior weights, false discovery penalties, group structures, and dependence assumptions?
• RQ3 Can a unified approach maintain strong FDR control while adapting to diverse and complex prior knowledge structures?
• RQ4 How does the inclusion of overlapping group partitions and dependence information affect the performance of multiple testing procedures?
• RQ5 What is the theoretical and practical advantage of combining all four types of prior knowledge in one algorithmic framework?

Key findings

• p-filter successfully unifies a wide range of existing multiple testing procedures as special cases by incorporating all four types of prior knowledge.
• The framework enhances statistical power and precision by leveraging heterogeneous prior information in a principled and integrated manner.
• By allowing simultaneous use of prior weights, false discovery penalties, group structures, and dependence assumptions, p-filter enables more adaptive and informed inference.
• The method maintains strong FDR control while offering greater flexibility than existing approaches that handle only subsets of prior knowledge.
• p-filter provides a scalable and general-purpose solution that can be applied across diverse scientific domains requiring multiple testing with complex prior information.