[Paper Review] PAPRIKA: Private Online False Discovery Rate Control

Wanrong Zhang, Gautam Kamath+1 more|arXiv (Cornell University)|Feb 27, 2020

Privacy-Preserving Technologies in Data14 references4 citations

TL;DR

PAPRIKA introduces a novel differentially private online algorithm for false discovery rate (FDR) control in multiple hypothesis testing, enabling real-time decisions with strong privacy and statistical guarantees. It achieves state-of-the-art performance by combining private thresholding with adaptive wealth management, significantly outperforming naive privatization methods in FDR control and statistical power under ε-differential privacy.

ABSTRACT

In hypothesis testing, a false discovery occurs when a hypothesis is incorrectly rejected due to noise in the sample. When adaptively testing multiple hypotheses, the probability of a false discovery increases as more tests are performed. Thus the problem of False Discovery Rate (FDR) control is to find a procedure for testing multiple hypotheses that accounts for this effect in determining the set of hypotheses to reject. The goal is to minimize the number (or fraction) of false discoveries, while maintaining a high true positive rate (i.e., correct discoveries). In this work, we study False Discovery Rate (FDR) control in multiple hypothesis testing under the constraint of differential privacy for the sample. Unlike previous work in this direction, we focus on the online setting, meaning that a decision about each hypothesis must be made immediately after the test is performed, rather than waiting for the output of all tests as in the offline setting. We provide new private algorithms based on state-of-the-art results in non-private online FDR control. Our algorithms have strong provable guarantees for privacy and statistical performance as measured by FDR and power. We also provide experimental results to demonstrate the efficacy of our algorithms in a variety of data environments.

Motivation & Objective

• To address the challenge of controlling false discovery rate (FDR) in online multiple hypothesis testing while ensuring differential privacy for individual data points.
• To design a private online algorithm that maintains high statistical power and strong FDR control, even when hypotheses are tested sequentially and decisions must be made immediately.
• To overcome the limitations of prior offline private FDR methods that require all p-values upfront, which is impractical in streaming or adaptive data analysis settings.
• To provide provable guarantees on both privacy (ε-differential privacy) and statistical performance (FDR and power) in the online setting.
• To empirically validate the algorithm’s robustness and accuracy across diverse data distributions, including Bernoulli and truncated exponential models.

Proposed method

• The method extends non-private online FDR procedures (e.g., SAFFRON) by incorporating differential privacy using the ReportNoisyMin mechanism and a private thresholding strategy.
• It introduces a private wealth-based mechanism that dynamically adjusts rejection thresholds based on accumulated privacy loss and observed p-values.
• A shift parameter A is introduced to calibrate the private thresholding process, balancing FDR control and statistical power.
• The algorithm uses a private version of the Benjamini-Hochberg procedure, where noisy p-values are processed through a sequence of private comparisons to determine rejections.
• Theoretical analysis bounds the expected FDR as α + δt, where δ is the privacy loss parameter, ensuring that FDR remains within a target level.
• The method applies post-processing invariance and advanced composition theorems to maintain privacy across sequential queries.

Experimental results

Research questions

• RQ1 Can a differentially private online FDR control algorithm be designed that maintains strong statistical performance while ensuring privacy for individual data points?
• RQ2 How does the performance of private online FDR control compare to non-private and naively privatized baselines in terms of FDR and statistical power?
• RQ3 What is the impact of the shift parameter A on the trade-off between FDR control and statistical power in the private online setting?
• RQ4 Can private online FDR control be effectively applied to real-world data distributions such as Bernoulli and truncated exponential models?
• RQ5 How do private thresholding and wealth management mechanisms compare to naive privatization (e.g., Laplace noise on p-values) in terms of FDR and power?

Key findings

• PAPRIKA achieves FDR control within the target level α, with empirical FDR bounded by α + δt, demonstrating strong theoretical privacy guarantees.
• In experiments with Bernoulli and truncated exponential data, PAPRIKA maintains high statistical power (up to 80% under strong signal) while keeping FDR below 5% for ε=5.
• The algorithm outperforms LapSAFFRON—a naive privatization of SAFFRON using Laplace noise—significantly in both FDR control and power, especially under weak signals (θi=1.90).
• The choice of shift parameter s (related to A) critically affects the FDR-power trade-off: larger s values reduce rejections, improving FDR but reducing power.
• Wealth and rejection threshold trajectories in PAPRIKA closely mirror those of non-private SAFFRON, confirming that private decisions align with optimal non-private decisions.
• PAPRIKA AI, which uses adaptive λj=αj, shows improved performance over PAPRIKA with fixed λj=0.2, particularly in high-signal regimes (θi=2.00).